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ABSTRACT 



Apparatus and methods are disclosed for simultaneously 
tracking multiple finger and palm contacts as hands 
approach, touch, and slide across a proximity-sensing, 
compliant, and flexible multi-touch surface. The surface 
consists of compressible cushion, dielectric, electrode, and 
circuitry layers. A simple proximity transduction circuit is 
placed under each electrode to maximize signal-to-noise 
ratio and to reduce wiring complexity. Such distributed 
transduction circuitry is economical for large surfaces when 
implemented with thin-film transistor techniques. Scanning 
and signal oflset removal on an electrode array produces 
low-noise proximity images. Segmentation processing of 
each proximity image constructs a group of electrodes 
corresponding to each distinguishable contact and extracts 
shape, position and surface proximity features for each 
group. Groups in successive images which correspond to the 
same hand contact are linked by a persistent path tracker 
which also detects individual contact touchdown and liftoff. 
Combinatorial optimization modules associate each con- 
tact's path with a particular fingertip, thumb, or palm of 
either hand on the basis of biomechanical constraints and 
contact features. Classification of intuitive hand configura- 
tions and motions enables unprecedented integration of 
typing, resting, pointing, scrolling, 3D manipulation, and 
handwriting into a versatile, ergonomic computer input 
device. 

118 Claims, 45 Drawing Sheets 
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METHOD AND APPARATUS FOR 
INTEGRATING MANUAL INPUT 

The present application is based upon provisional appli- 
cation Serial No. 60/072,509, filed Jan. 26, 1998. 

BACKGROUND OF THE INVENTION 

A. Field of the Invention 

The present invention relates generally to methods and 
apparatus for data input, and, more particularly, to a method 
and apparatus for integrating manual input. 

B. Description of the Related Art 

Many methods for manual input of data and commands to 
computers are in use today, but each is most efficient and 
easy to use for particular types of data input. For example, 
drawing tablets with pens or pucks excel at drafting, 
sketching, and quick command gestures. Handwriting with 
a stylus is convenient for filling out forms which require 
signatures, special symbols, or small amounts of text, but 
handwriting is slow compared to typing and voice input for 
long documents. Mice, finger-sticks and touchpads excel at 
cursor pointing and graphical object manipulations such as 
drag and drop. Rollers, thumbwheels and trackballs excel at 
panning and scrolling. The diversity of tasks that many 
computer users encounter in a single day call for all of these 
techniques, but few users will pay for a multitude of input 
devices, and the separate devices are often incompatible in 
a usability and an ergonomic sense. For instance, drawing 
tablets are a must for graphics professionals, but switching 
between drawing and typing is inconvenient because the pen 
must be put down or held awkwardly between the fingers 
while typing. Thus, there is a long-felt need in the art for a 
manual input device which is cheap yet offers convenient 
integration of common manual input techniques. 

Speech recognition is an exciting, new technology which 
promises to relieve some of the input burden on user hands. 
However, voice is not appropriate for inputting all types of 
data either. Currently, voice input is best-suited for dictation 
of long text documents. Until natural language recognition 
matures sufficiently that- very high, level .voice commands 
can be understood by the computer, voice will have little 
advantage over keyboard hot-keys and mouse menus for 
command and control. Furthermore, precise pointing, 
drawing, and manipulation of graphical objects is difficult 
with voice commands, no matter how well speech is under- 
stood. Thus, there will always be a need in the art for 
multi-function manual input devices which supplement 
voice input. 

A generic manual input device which combines the 
typing, pointing, scrolling, and handwriting capabilities of 
the standard input device collection must have ergonomic, 
economic, and productivity advantages which outweigh the 
unavoidable sacrifices of abandoning device specialization. 
The generic device must tightly integrate yet clearly distin- 
guish the different types of input. It should therefore appear 
modeless to the user in the sense that the user should not 
need to provide explicit mode switch signals such as 
buttonpresses, arm relocations, or stylus pickups before 
switching from one input activity to another Epidemiologi- 
cal studies suggest that repetition and force multiply in 
causing repetitive strain injuries. Awkward postures, device 
activation force, wasted motion, and repetition should be 
minimized to improve ergonomics. Furthermore, the work- 
load should be spread evenly over all available muscle 
groups to avoid repetitive strain. 

Repetition can be minimized by allocating to several 
graphical manipulation channels those tasks which require 
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complex mouse pointer motion sequences. Common graphi- 
cal user interface operations such as finding and manipulat- 
ing a scroll bar or slider control are much less efficient than 
specialized finger motions which cause scrolling directly, 

5 without the step of repositioning the cursor over an 
on-screen control. Preferably the graphical manipulation 
channels should be distributed amongst many finger and 
hand motion combinations to spread the workload. Touch- 
pads and mice with auxilliary scrolling controls such as the 

io Cirque® Smartcat touchpad with edge scrolling, the IBM® 
ScrollPoint™ mouse with embedded pointing stick, and the 
Roller Mouse described in U.S. Pat. No. 5,530,455 to Gillick 
et al. represent small improvements in this area, but still do 
not provide enough direct manipulation channels to elimi- 

15 nate many often-used cursor motion sequences. 
Furthermore, as S. Zhai et al. found in "Dual Stream Input 
for Pointing and Scrolling," Proceedings of CHI *97 
Extended Abstracts (1997), manipulation of more than two 
degrees of freedom at a time is very difficult with these 

20 devices, preventing simultaneous panning, zooming and 
rotating. 

Another common method for reducing excess motion and 
repetition is to automatically continue pointing or scrolling 
movement signals once the user has stopped moving or lifts 

25 the finger. Related art methods can be distinguished by the 
conditions under which such motion continuation is enabled. 
In U.S. Pat. No. 4,734,685, Watanabe continues image 
panning when the distance and velocity of pointing device 
movement exceed thresholds. Automatic panning is stopped 

30 by moving the pointing device back in the opposite 
direction, so stopping requires additional precise move- 
ments. In U.S. Pat. No. 5,543,591 to Gillespie et al., motion 
continuation occurs when the finger enters an edge border 
region around a small touchpad. Continued motion speed is 

35 fixed and the direction corresponds to the direction from the 
center of the touchpad to the finger at the edge. Continuation 
mode ends when the finger leaves the border region or lifts 
off the pad. Disadvantageously, users sometimes pause at the 
edge of the pad without intending for cursor motion to 

40 continue, and the unexpected motion continuation becomes 
annoying. U.S. Pat. No. 5,327,161 to Logan et al. describes 
motion continuation when the ringer enters a border area as 
well, but in an alternative trackball emulation mode, motion 
continuation can be a function solely of lateral finger veloc- 

45 ity and direction at liftoff. Motion continuation decays due 
to a friction factor or can be stopped by a subsequent 
touchdown on the surface. Disadvantageously, touch veloc- 
ity at liftoff is not a reliable indicator of the user's desire for 
motion continuation since when approaching a large target 

50 on a display at high speeds the user may not stop the pointer 
completely before liftoff. Thus it would be an advance in the 
art to provide a motion continuation method which does not 
become activated unexpectedly when the user really 
intended to stop pointer movement at a target but happens to 

55 be on a border or happens to be moving at significant speed 
during liftoff. 

Many attempts have been made to embed pointing 
devices in a keyboard so the hands don't have to leave 
typing position to access the pointing device. These include 

60 the integrated pointing key described in U.S. Pat No. 
5,189,403 to Franz et al., the integrated pointing stick 
disclosed by J. Rutledge and T. Selker in " Force -to-Motion 
Functions for Pointing/* Human-Computer Interaction- 
INTERACT '90, pp. 701-06 (1990), and the position sensing 

65 keys described in U.S. Pat. No. 5,675,361 to Santilli. 
Nevertheless, the limited movement range and resolution of 
these devices leads to poorer pointing speed and accuracy 
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than a mouse, and they add mechanical complexity to need in the art tor a multi-touch surface typing system based 

keyboard construction. Thus there exists a need in the art far on zero-force capacitive sensing which can tolerate resting 

pointing methods with higher resolution, larger movement hands and a surface cushion. 

range, and more degrees of freedom yet which are easily An ergonomic typing system should also adapt to iodi- 

accessible from typing hand positions. 5 vidual hand sizes, tolerate variations in typing style, and 

Touch screens and touchpads often distinguish pointing support a range of healthy hand postures. Though many 
motions from emulated button clicks or keypresses by ergonomic keyboards have been proposed, mechanical key- 
assuming very little lateral fingertip motion will occur switches can only be repositioned at great cost. For example, 
during taps on the touch surface which are intended as the keyboard with concave keywells described by Har- 
clicks. Inherent in these methods is the assumption that greaves et al. in U.S. Pat. No. 5,689^53 fits most hands well 
tapping will usually be straight down from the suspended but also tends to lock the arms in a single position. A touch 
finger position, minimizing those components of finger surface key layout could easily be raorpbed, translated, or 
motion tangential to the surface. This is a valid assumption arbitrarily reconfigured as long as the changes didn't con- 
if the surface is not finely divided into distinct key areas or fuse the user. However, touch surfaces may not provide as 
if the user does a slow, "bunt and peck*' visual search for 15 much laterally orienting tactile feedback as the edges of 
each key before striking. For example, in U.S. Pat. No. mechanical keyswitches. Thus, there exists a need in the art 
5,543,591 to Gillespie et al. a touchpad sends all lateral for a surface typing recognizer which can adapt a key layout 
motions to the host computer as cursor movements. to fit individual hand postures and which can sustain typing 
However, if the finger is lifted soon enough after touchdown accuracy if the hands drift due to limited tactile feedback, 
to count as a tap and if the accumulated lateral motions are 2 o Handwriting on smooth touch surfaces using a stylus is 
not excessive, any sent motions are undone and a mouse well-known in the art, but it typically doesn't integrate well 
button click is sent instead. This method only works for with typing and pointing because the stylus must be put 
mouse commands such as pointing which can safely be down somewhere or held awkwardly during other input 
undone, not for dragging or other manipulations. In U.S. Pat. activities. Also, it may be difficult to distinguish the hand- 
No. 5,666,113 to Logan, taps with less than about Vie" lateral 25 writing activity of the stylus from pointing motions of a 
motion activate keys on a small keypad while lateral motion fingertip. Thus there exists a need in the art for a method to 
in excess of Vie" activates cursor control mode. In both capture coarse handwriting gestures without a stylus and 
patents cursor mode is invoked by default when a finger without confusing them with pointing motions, 
stays on the surface a long time. Many of the input differentiation needs cited above could 

However, fast touch typing on a surface divided into a 30 be met with a touch sensing technology which distinguishes 

large array of key regions tends to produce more tangential a variety of hand configurations and motions such as sliding 

motions along the surface than related art filtering tech- finger chords and grips. Many mechanical chord keyboards 

niques can tolerate. Such an array contains keys in multiple have been designed to detect simultaneous downward activ- 

rows and columns which may not be directly under the ity from multiple fingers, but they do not detect lateral finger 

fingers, so the user must reach with the hand or flex or 35 motion over a large range. Related art shows several 

extend fingers to touch many of the key regions. Quick examples of capacitive touchpads which emulate a mouse or 

reaching and extending imparts significant lateral finger keyboard by tracking a single finger. These typically mea- 

motion while the finger is in the air which may still be sure the capacitance of or between elongated wires which 

present when the finger contacts the surface. Glancing taps are laid out in rows and columns. A thin dielectric is 

with as much as Vi" lateral motion measured at the surface 40 interposed between the row and column layers. Presence of- 

can easily result. Attempting to filter or suppress this much a finger perturbs the self or mutual capacitance for nearby 

morion would make the cursor seem sluggish and unrespon- electrodes. Since most of these technologies use projective 

sive. Furthermore, it may be desirable to enter a typematic row and column sensors which integrate on one electrode 

or automatic key repeat mode instead of pointing mode the proximity of all objects in a particular row or column, 

when the finger is held in one place on the surface. Any 4s they cannot uniquely determine the positions of two or more 

lateral shifting by the fingertip during a prolonged finger objects, as discussed in S. Lee, "A Fast Multiple-Touch- 

press would also be picked up as cursor jitter without heavy Sensitive Input Device" University of Toronto Masters 

filtering. Thus, there is a need in the art for a method to Thesis (1984). The best they can do is count fingertips which 

distinguish keying from pointing on the same surface via happen to lie in a straight row, and even that will fail if a 

more robust hand configuration cues than lateral motion of 50 thumb or palm is introduced in the same column as a 

a single finger. fingertip. 

An ergonomic typing system should require minimal key In U.S. Pat. Nos. 5,565,658 and 5,305,017, Gerpheide et 

tapping force, easily distinguish finger taps from resting al. measure the mutual capacitance between row and column 

hands, and cushion the fingers from the jarring force of electrodes by driving one set of electrodes at some clock 

surface impact Mechanical and membrane keyboards rely 55 frequency and sensing how much of that frequency is 

on the spring force in the keyswitches to prevent activation coupled onto a second electrode set. Such synchronous 

when the hands are resting on the keys. This causes an measurements are very prone to noise at the driving 

irreconcilable tradeoff between the ergonomic desires to frequency, so to increase signal-to-noise ratio they form 

reduce the fatigue from key activating force and to relax the virtual electrodes comprised of multiple rows or multiple 

full weight of the hands onto the keys during rest periods. 60 columns, instead of a single row and column, and scan 

Force minimization on touch surfaces is possible with through electrode combinations until the various mutual 

capacitive or active optical sensing, which do not rely on capacitances are nulled or balanced The coupled signal 

finger pressure, rather than resistive-membrane or surface- increases with the product of the rows and columns in each 

acoustic-wave sensing techniques. The related art touch virtual electrode, but the noise only increases with the sum, 

devices discussed below will become confused if a whole 65 giving a net gain in signal-to-noise ratio for virtual elec- 

hand, including its four fingertips, a thumb and possibly trades consisting of more than two rows and two columns, 

palm heels, rests on the surface. Thus, there exists a long felt However, to uniquely distinguish multiple objects, virtual 
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electrode sizes would have to be reduced so the intersection 
of the row and column virtual electrodes would be no larger 
than a finger tip, i.e. about two rows and two columns, which 
will degrade the signal-to-noise ratio. Also, the signal-to- 
noise ratio drops as row and column lengths increase to 
cover a large area. 

In U.S. Pat. Nos. 5,543,591, 5,543,590, and 5,495,077, 
Gillespie et al measure the electrode-ringer self-capacitance 
for row and column electrodes independently. Total elec- 
trode capacitance is estimated by measuring the electrode 
voltage change caused by injecting or removing a known 
amount of charge in a known time. All electrodes can be 
measured simultaneously if each electrode has its own 
drive/sense circuit. The centroid calculated from all row and 
column electrode signals establishes an interpolated vertical 
and horizontal position for a single object. This method may 
in general have higher signal-to-noise ratio than synchro- 
nous methods, but the signal-to-noise ratio is still degraded 
as row and column lengths increase. Signal-to-noise ratio is 
especially important for accurately locating objects which 
are floating a few millimeters above the pad. Though this 
method can detect such objects, it tends to report their 
position as being near the middle of the pad, or simply does 
not detect floating objects near the edges. 

Thus there exists a need in the art for a capacitance- 
sensing apparatus which does not suffer from poor signal- 
to-noise ratio and the multiple finger indistinguishability 
problems of touchpads with long row and column elec- 
trodes. 

U.S. Pat No. 5,463,388 to Boie et al. has a capacitive 
sensing system applicable to either keyboard or mouse 
input, but does not consider the problem of integrating both 
types of input simultaneously. Though they mention inde- 
pendent detection of arrayed unit-cell electrodes, their 
capacitance transduction circuitry appears too complex to be 
economically reproduced at each electrode. Thus the long 
lead wires connecting electrodes to remote signal condition- 
ing circuitry can pickup noise and will have significant 
capacitance compared to the finger-electrode self- 
capacitance, again limiting signal-to-noise ratio. Also, they 
do not recognize the importance of independent electrodes 
for multiple finger tracking, or mention how to track mul- 
tiple fingers on an independent electrode array. 

Lee built an early multi-touch electrode array with 7 mm 
by 4 mm metal electrodes arranged in 32 rows and 64 
columns. The "Fast Multiple-Touch-Sensitrve Input Device 
(FMTSID)" total active area measured 12" by 16", with a 
0.075 mm Mylar dielectric to insulate fingers from elec- 
trodes. Each electrode had one diode connected to a row 
charging line and a second diode connected to a column 
discharging line. Electrode capacitance changes were mea- 
sured singly or in rectangular groups by raising the voltage 
on one or more row lines, selectively charging the electrodes 
in those rows, and then timing the discharge of selected 
columns to ground through a discharge resistor. Lee's design 
required only two diodes per electrode, but the principal 
disadvantage of Lee's design is that the column diode 
reverse bias capacitances allowed interference between elec- 
trodes in the same column. 

All of the related capacitance sensing art cited above 
utilize interpolation between electrodes to achieve high 
pointing resolution with economical electrode density. Both 
Boie et al. and Gillespie et al. discuss computation of a 
centroid from all row and column electrode readings. 
However, for multiple finger detection, centroid calculation 
must be carefully limited around local maxima to include 
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only one finger at a time. Lee utilizes a bisective search 
technique to find local maxima and then interpolates only on 
the eight nearest neighbor electrodes of each local maximum 
electrode. This may work fine for small fingertips, but thumb 

5 and palm contacts may cover more than nine electrodes. 
Thus there exists a need in the art for improved means to 
group exactly those electrodes which arc covered by each 
distinguishable hand contact and to compute a centroid from 
such potentially irregular groups. 

10 To take maximum advantage of multi-touch surface 
sensing, complex proximity image processing is necessary 
to track and identify the parts of the hand contacting the 
surface at any one time. Compared to passive optical 
images, proximity images provide clear indications of where 

15 the body contacts the surface, uncluttered by luminosity 
variation and extraneous objects in the background. Thus 
proximity image filtering and segmentation stages can be 
simpler and more reliable than in computer vision 
approaches to free-space hand tracking such as S. Ahmad, 

20 "A Usable Real-Time 3D Hand Tracker", Proceedings of the 
28 th Asilomar Conference on Signals, Systems, and 
Computers-Part 2, vol. 2, IEEE (1994) or Y Cui and J. 
Wang, "Hand Segmentation Using Learning-Based Predic- 
tion and Verification for Hand Sign Recognition," Proceed- 

25 ings of the 1996 IEEE Computer Society Conference on 
Computer Vision and Pattern Recognition, pp. 88-93 
(1996). However parts of the hand such as intermediate 
finger joints and the center of the palms do not show up in 
capacitive proximity images at all if the hand is not flattened 

30 on the surface. Without these intermediate linkages between 
fingertips and palms the overall hand structure can only be 
guessed at, making hand contact identification very difficult. 
Hence the optical flow and contour tracking techniques 
which have been applied to free-space hand sign language 

35 recognition as in F. Quek, "Unencumbered Gestural 
Interaction," IEEE Multimedia, vol. 3, pp. 36-47 (1996), do 
not address the special challenges of proximity image track- 
ing. 

Synaptics Corp. has successfully fabricated their elec- 
40 trbde array on flexible mylar film rather than stiff circuit 
board. This is suitable for conforming to the contours of 
special products, but does not provide significant finger 
cushioning for large surfaces. Even if a cushion was placed 
under the film, the lack of stretchability in the film, leads, 
45 and electrodes would limit the compliance afforded by the 
compressible material. Boie et al suggests that placing 
compressible insulators on top of the electrode array cush- 
ions finger impact. However, an insulator more than about 
one millimeter thick would seriously attenuate the measured 
50 finger-electrode capacitances. Thus there exists a need in the 
art for a method to transfer finger capacitance influences 
through an arbitrarily thick cushion. 

SUMMARY OF THE INVENTION 

55 

It is a primary object of the present invention to provide 
a system and method for integrating different types of 
manual input such as typing, multiple degree-of- freedom 
manipulation, and handwriting on a multi-touch surface. 

60 It is also an object of the present invention to provide a 
system and method for distinguishing different types of 
manual input such as typing, multiple de gree -o f- f re e do m 
manipulation, and handwriting on a multi-touch surface, via 
different hand confiourations which are easy for the user to 

55 learn and easy for the system to recognize. 

It is a further object of the present invention to provide an 
improved caparitance-transducing apparatus that is cheaply 
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implemented near each electrode so that two-dimensional an output node; a dielectric-covered sensing electrode con- 
sensor arrays of arbitrary size and resolution can be built nected to the common node between the two switching 
without degradation in signal to noise. means; a power supply providing an approximately constant 
It is a further object of the present invention to provide an volta SP connected to the input node of the series-connected 
electronic system which minimizes the number of sensing 5 switching means; and an integrating current-to-voltage 
electrodes necessary to obtain proximity images with such device connected to the output node of the series 
resolution that a variety of hand configurations can be ^ 0nn ? ctcd SWlt ' hm * mcan f; mc current-to-voltage transla- 
distinffuished device producing a voltage representing the magnitude 

. of the self-capacitance of the sensing device. 

Yet another object of the present invention is to provide a T A ... . • 

tA . A , * 5 . ■ . - , 10 lo further achieve the objects, the present invention 

multi-touch surface apparatus which is compliant and con- ■ „ „ „ . u - - . , c „ . c , . ■ 

* , . ^ l i , - 1 * , comprises a multi-touch surtace apparatus for detecting a 

toured to be comfortable and ergonomic under extended use. ** i _ * r u- i * u j ■ Ti_ 

& spatial arrangement of multiple touch devices on or near the 

Yet another object of the present invention is to provide surface of the multi-touch apparatus, comprising: one of a 

tactile key or hand position feedback without impeding hand rigid or flexible surface; a plurality of two-dimensional 

resting on the surface or smooth, accurate sliding across the* JS arrays G f oae 0 f the sensing devices (recited in the previous 

surface. paragraph) arranged on the surface in groups wherein the 

It is a further object of the present invention to provide an sensing devices within a group have their output nodes 

electronic system which can provide images of flesh prox- connected together and share the same integrating capacitor, 

imity to an array of sensors with such resolution that a charge depletion switch, and voltage -to- voltage translation 

variety of hand configurations can be distinguished. 2 0 circuitry; control circuitry for enabling a single sensor 

It is another object of the present invention to provide an device from each two-dimensional array; means for select- 
improved method for invoking cursor motion continuation m fi tnc sensor voltage data from each two-dimensional 
only when the user wants it by not invoking it when array; voltage measurement circuitry to convert sensor volt- 
significant deceleration is detected. a ge data to a digital code; and circuitry for communicating 

Another object of the present invention is to identify 25 the digital code to another electronic device. The sensor 
different hand parts as tbey contact the surface so that a volta g e data selecting means comprises one of a multiplex- 
variety of hand configurations can be recognized and used to m 8 cucuitrv and a P luraUtv of voltage measurement circuits, 
distinguish different kinds of input activity. To stm achieve the objects, the present invention 

Yet another object of the present invention is to reliably ! ^onch surface apparatus for sensing 

^ » j ,• „ 4 t . 30 diverse configurations and activities of touch devices and 

extract rotation and scaling as well as translation degrees of . & , . . - 

r , _ r c . , , * t\ generating integrated manual input to one of an electronic or 

freedom from the motion of two or more hand contacts to aid , _ ° , ® . , . r 

. , • i *■ c *. i . . , . electro -mechanical device, the apparatus comprising: an 

in navigation and manipulation of two-dimensional elec- P ~ , * . vv . , . v , & , 

tronic documents. array 0 one of me P roximit y sensmg devices described 

- L . . above; a dielectric cover having symbols printed thereon 

It is a further object of the present invention to reliably 3S mat rcpTCSCnt action-to-be-taken when engaged by the touch 

extract tilt and roll degrees of freedom from hand pressure devices; mmdng meaos for forming digital p roximity 

differences to aid in navigation and manipulation of three- from the ^ of devices; calibrating means 

dimensional environments. for removmg background of^ts from me proximity images; 

Additional objects and advantages of the invention will be recognition means for interpreting the configurations and 

set forth in part in the description which follows, and. in part 40 activities of the touch devices that make up the proximity 

will be obvious from the description, or may be learned by images; processing means for generating input signals in 

practice of the invention. The objects and advantages of the response to particular touch device configurations and 

invention will be realized and attained by means of the motions; and communication means for sending the input 

elements and combinations particularly pointed out in the signals to the electronic or electro-mechanical device, 

appended claims. 45 j Q even further achieve the objects, the present invention 

To achieve the objects and in accordance with the purpose comprises a multi-touch surface apparatus for sensing 
of the invention, as embodied and broadly described herein, diverse configurations and activities of fingers and palms of 
the invention comprises a sensing device that is sensitive to one or more hands near the surface and generating integrated 
changes in self-capacitance brought about by changes in manual input to one of an electronic or electro-mechanical 
proximity of a touch device to the sensing device, the 50 device, the apparatus comprising: an array of proximity 
sensing device comprising: two electrical switching means sensing means embedded in the surface; scanning means for 
connected together in series having a common node, an forming digital proximity images from the proximities mea- 
input node, and an output node; a dielectric-covered sensing sured by the sensing means; image segmentation means for 
electrode connected to the common node between the two collecting into groups those proximity image pixels inten- 
s witching means; a power supply providing an approxi- 55 sified by contact of the same distinguishable part of a hand; 
mately constant voltage connected to the input node of the contact tracking means for parameterizing hand contact 
series-connected switching means; an integrating capacitor features and trajectories as the contacts move across sue- 
to accumulate charge transferred during multiple consecu- ccssive proximity images: contact identification means for 
tive switching of the series connected switching means; determining which hand and which part of the band is 
another switching means connected in parallel across the 60 causing each surface contact; synchronization detection 
integrating capacitor to deplete its residual charge; and a means for identifying subsets of identified contacts which 
voltage-to-voltage translation device connected to the output touchdown or liftoff the surface at approximately the same 
node of the series-connected switching means which pro- time, and for generating command signals in response to 
duces a voltage representing the magnitude of the self- synchronous taps of multiple fingers on the surface; typing 
capacitance of the sensing device. Alternatively, the sensing 65 recognition means for generating intended key symbols 
device comprises: two electrical switching means connected from asynchronous finger taps; motion component extrac- 
together in series having a common node, an input node, and tion means for compressing multiple degrees of freedom of 
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multiple fingers into degrees of freedom common in two and 
three dimensional graphical manipulation; chord motion 
recognition means for generating one of command and 
cursor manipulation signals in response to motion in one or 
more extracted degrees of freedom by a selected combina- 5 
tion of fingers; pen grip detection means for recognizing 
contact arrangements which resemble the configuration of 
the hand when gripping a pen, generating inking signals 
from motions of the inner fingers, and generating cursor 
manipulation signals from motions of the palms while the 10 
inner fingers are lifted; and communication means for send- 
ing the sensed configurations and activities of finger and 
palms to one of the electronic and electro-mechanical 
device. 

To further achieve the objects, the present invention ^ 
comprises a method for tracking and identifying hand con- 
tacts in a sequence of proximity images in order to support 
interpretation of hand configurations and activities related to 
typing, multiple degree -of -freedom manipulation via 
chords, and handwriting, the method comprising the steps 2 o 
of: segmenting each proximity image into groups of elec- 
trodes which indicate significant proximity, each group 
representing proximity of a distinguishable hand part or 
other touch device; extracting total proximity, position, 
shape, size, and orientation parameters from each group of 25 
electrodes; tracking group paths through successive prox- 
imity images including detection of path endpoints at con- 
tact touchdown and liftoff; computing velocity and filtered 
position vectors along each path; assigning a hand and finger 
identity to each contact path by incorporating relative path 30 
positions and velocities, individual contact features, and 
previous estimates of hand and finger positions; and main- 
taining estimates of hand and finger positions from trajec- 
tories of paths currently assigned to the fingers, wherein the 
estimates provide high level feedback to bias segmentations 35 
and identifications in future images. 

To still further achieve the objects, the present invention 
comprises a method for integrally extracting multiple 
degrees of freedom of hand motion from sliding motions of 
two or more fingers of a hand across a multi-touch surface, 40 
one of the fingers preferably being the opposable thumb, the 
method comprising the steps of: tracking across successive 
scans of the proximity sensor array the trajectories of 
individual hand parts on the surface; finding an innermost 
and an outermost finger contact from contacts identified as 45 
fingers on the given hand; computing a scaling velocity 
component from a change in a distance between the inner- 
most and outermost finger contacts; computing a rotational 
velocity component from a change in a vector angle between 
the innermost and outermost finger contacts; computing a 50 
translation weighting for each contacting finger; computing 
translational velocity components in two dimensions from a 
translation weighted average of the finger velocities tangen- 
tial to surface; suppressively filtering components whose 
speeds are consistently lower than the fastest components; 55 
transmitting the filtered velocity components as control 
signals to an electronic or electro-mechanical device. 

To even further achieve the objects, the present invention 
comprises a manual input integration method for supporting 
diverse hand input activities such as resting the hands, 60 
typing, multiple degree-of-freedom manipulation, command 
gesturing and handwriting on a multi-touch surface, the 
method enabling users to instantaneously switch between 
the input activities by placing their bands in different con- 
figurations comprising distinguishable combinations of rela- 65 
tive hand contact timing, proximity, shape, size, position, 
motion and/or identity across a succession of surface prox- 
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imity images, the method comprising the steps of: tracking 
each touching hand part across successive proximity images; 
measuring the times when each hand part touches down and 
lifts off the surface; detecting when hand parts touch down 
or lift off simultaneously; producing discrete key symbols 
when the user asynchronously taps, holds, or slides a finger 
on kcyregions defined on the surface; producing discrete 
mouse button click commands, key commands, or no signals 
when the user synchronously taps two or more fingers from 
the same hand on the surface; producing gesture commands 
or multiple degree-of-freedom manipulation signals when 
the user slides two or more fingers across the surface; and 
sending the produced symbols, commands and manipulation 
signals as input to an electronic or an electro- mechanical 
device. 

To still even further achieve the objects, the present 
invention comprises a method for choosing what kinds of 
input signals will be generated and sent to an electronic or 
electro-mechanical device in response to tapping or sliding 
of fingers on a multi-touch surface, the method comprising 
the following steps: identifying each contact on the surface 
as either a thumb, fingertip or palm; measuring the times 
when each hand part touches down and lifts off the surface; 
forming a set of those lingers which touch down from the all 
finger floating state before any one of the fingers lifts back 
off the surface; choosing the kinds of input signals to be 
generated by further distinctive motion of the fingers from 
the combination of finger identities in the set; generatinng 
input signals of this kind when further distinctive motions of 
the fingers occur; forming a subset any two or more fingers 
which touch down synchronously after at least one finger 
has lifted back off the surface; choosing a new kinds of input 
signals to be generated by further distinctive motion of the 
fingers from the combination of finger identities in the 
subset; generating input signals of this new kind when 
further distinctive motions of the fingers occur; and con- 
tinuing to form new subsets, choose and generate new kinds 
of input signals in response to liftoff and synchronous 
touchdowns until all fingers lift off the surface. 

To further achieve the objects, the present invention 
comprises a- method for continuing- generation of cursor 
movement or scrolling signals from a tangential motion of a 
touch device over a touch-sensitive input device surface 
after touch device liftoff from the surface if the touch device 
operator indicates that cursor movement continuation is 
desired by accelerating or failing to decelerate the tangential 
motion of the touch device before the touch device is lifted, 
the method comprising the following steps: measuring, 
storing and transmitting to a computing device two or more 
representative tangential velocities during touch device 
manipulation; computing and storing a liftoff velocity from 
touch device positions immediately prior to the touch device 
liftoff; comparing the liftoff velocity with the representative 
tangential velocities, and entering a mode for continuously 
moving the cursor if a tangential liftoff direction approxi- 
mately equals the representative tangential directions and a 
tangential liftoff speed is greater than a predetermined 
fractional multiple of representative tangential speeds; con- 
tinuously transmitting cursor movement signals after liftoff 
to a computing device such that the cursor movement 
velocity corresponds to one of the representative tangential 
velocities; and ceasing transmission of the cursor movement 
signals when the touch device engages the surface again, if 
comparing means detects significant deceleration before 
liftoff, or if the computing device replies that the cursor can 
move no farther or a window can scroll 00 farther. 

It is to be understood that both the foregoing general 
description and the following detailed description are exem- 
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plary and explanatory only and are not restrictive of the Fl G. 21 is a flow chart of segmentation edge testing; 

invention, as claimed. FIG. 22 is a flow chart of persistent path tracking; 

BRIEF DESCRIPTION OF THE DRAWINGS , 23 * a flow chart of thc hand identification 

algorithm; 

The accompanying drawings, which are incorporated in 5 FIG. 24 is a Vbronoi cell diagram constructed around hand 

and constitute a part of this specification, illustrate several part attractor points; 

embodiments of the invention and together with the descrip- fj G 25A is a plot of orientation weighting factor for right 

tion serve to explain the principles of the invention. In the thumb, right inner palm, and left outer palm versus contact 

drawings: orientation; 

FIG. 1 is a block diagram of the integrated manual input FIG. 25B is a plot of thumb size factor versus contact size; 

apparatus; FIG. 25C is a plot of palm size factor versus ratio of total 

FIG. 2 is a schematic drawing of the proximity sensor contact proximity to contact eccentricity; 

with voltage amplifier; p, G 2 $D is a plot of palm separation factor versus 

FIG. 3 is a schematic drawing of the proximity sensor 15 distance between a contact and its nearest neighbor contact; 

with integrating current amplifier, nG 2 6 is a flow chart of the thumb presence verification 

FIG. 4 is a schematic drawing of the proximity sensor algorithm; 

implemented with field effect transistors; piG. 27 is a flow chart of an alternative hand part 

FIG. 5 is a schematic drawing of the proximity sensor as identification algorithm; 

used to implement 2D arrays of proximity sensors; 20 piG. 28 is a flow chart of the pen grip detection process; 

FIG. 6 is a block diagram showing a typical architecture FIG. 29 is a flow chart of the hand identification 

for a 2D array of proximity sensors where all sensors share algorithm, 

the same amplifier; F1GS 30A-C show three different hand partition hypoth- 

FIG. 7 is a block diagram of circuitry used to convert 25 eses for a fixed arrangement of surface contacts; 

proximity sensor output to a digital code; piG. 31A is a plot of the hand clutching direction factor 

FIG. 8 is a block diagram showing a typical architecture versus horizontal hand velocity; 

for a 2D array of proximity sensors where sensors within a FIG. 31B is a plot of the handedness factor versus vertical 

row share the same amplifier; position of outermost finger relative to next outermost; 

FIG. 9 is a schematic of a circuit useful for enabling the 30 pi G 3 1C ^ a p i ot Q f lQe pa i m cohesion factor versus 

output gates of all proximity sensors within a group maximum horizontal separation between palm contacts 

(arranged in columns); within a hand; 

FIG. 10 is a side view of a 2D proximity sensor array that FIG. 32 is a plot of the inner finger angle factor versus the 

is sensitive to the pressure exerted by non-conducting touch ang i e between the innermost and next innermost finger 

objects; 35 contacts; 

FIG. 11 is a side view of a 2D proximity sensor array that pi G . 33 ^ a p i ot c f t he inter-hand separation factor versus 

provides a compliant surface without loss of spatial sensi- the estimated distance between the right thumb and left 

tivity; thumb; 

. .^Pr^^*?*^^?^^?^.™^*^ 1 ™?^ 1 40 ... FIG.. 34 is a. flow chart. of hand, motion component, 

is sensitive to both the proximity of conducting touch extraction' 

objects and to the pressure exerted by non-conducting touch p, G ^ fc a diagram of typical finger trajectories when 

ob J ects; hand is contracting; 

FIG. 13 is an example proximity image of a hand flattened fig. 36 is a flow chart of radial and angular hand velocity 

onto the surface with fingers outstretched; 45 ex t rac ti 0 n- 

FIG. 14 is an example proximity image of a hand partially nG 3^ a flow chart sbowing extraction c f translaUonal 

closed with fingertips normal to surface; hand velocity components; 

FIG. 15 is an example proximity image of a hand in the nG 38 fe a fiow ^ of differ ential hand pressure 

pen grip configuration with thumb and index fingers extraction* 

I>U ^'^^* . , *. . . , . - * 50 FIG. 39A is a flow chart of the finger synchronization 

FIG. 16 is a data flow diagram of the hand tracking and detection loop- 
contact identification system; » ftT1 . ' „ . _ r . . . . 4 

nn „ w . n / * FIG. 39B is a flow chart of chord tap detection; 

FIG. 17 is a flow chart of hand position estimation; ^„ Mn A . „ , ^ r t . . . 

^ . , „ r „ ... FIG. 40 A is a flow chart of the chord motion recognition 

FIG. 18 is a data flow diagram of proximity image j 00 p. 

segmentation; nQ ^ {& a flow chaft of ch()rd moiion even( ra _ 

FIG. 19 is a diagram of the boundary search pattern tion' 

during construction of an electrode group; f ; g 41 ^ a flow ^ of fey layom morphing . 

FIG. 20A is a diagram of the segmentation strictness p, G 42 is a flow chart of thc keypress detection loop; 

regions with both hands in their neutral default position on M FIG. 43A is a flow chart of the keypress acceptance and 

surface; *_ • * 1 a 

transmission loop; and 

FIG. 20B is a diagram of the segmentation strictness mQ 433 ^ a flow chart of typematic emulation, 
regions when the hands are in asymmetric positions on 

surface; DESCRIPTION OF THE PREFERRED 

FIG. 20C is a diagram of the segmentation strictness 65 EMBODIMENTS 

regions when the right hand crosses to the left half of the Reference will now be made in detail to the present 

surface and the left hand is off the surface; preferred embodiments of the invention, examples of which 
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are illustrated in the accompanying drawings. Wherever 
possible, the same reference numbers will be used through- 
out the drawings to refer to the same or like parts. 

FIG. 1 is a system block diagram of the entire integrated 
manual input apparatus. Sensors embedded in the multi- 
touch surface 2 detect proximity of entire flattened hands 4, 
fingertips, thumbs, palms, and other conductive touch 
devices to the surface 2. In a preferred embodiment, the 
surface is large enough to comfortably accommodate both 
hands 4 and is arched to reduce forearm pronation. 

In alternative embodiments the multi-touch surface 2 may 
be large enough to accommodate motion of one hand, but 
may be flexible so it can be fitted to an armrest or clothing. 

Electronic scanning hardware 6 controls and reads from 
each proximity sensor of a sensor array. A calibration 
module 8 constructs a raw proximity image from a complete 
scan of the sensor array and subtracts off any background 
sensor offsets. The background sensor offsets can simply be 
a proximity image taken when nothing is touching the 
surface. 

The offset -corrected proximity image is then passed on to 
the contact tracking and identification module 10, which 
segments the image into distinguishable hand-surface 
contacts, tracks and identifies them as they move through 
successive images. 

The paths of identified contacts are passed on to a typing 
recognizer module 12, finger synchronization detection 
module 14, motion component extraction module 16, and 
pen grip detection module 17, which contain software algo- 
rithms to distinguish hand configurations and respond to 
detected hand motions. 

The typing recognizer module 12 responds to quick 
presses and releases of fingers which are largely asynchro- 
nous with respect to the activity of other fingers on the same 
hand. It attempts to find the key region nearest to the location 
of each finger tap and forwards the key symbols or com- 
mands associated with the nearest key region to the com- 
munication interface module 20. 

The finger synchronization detector 14 checks the finger 
activity within a hand for simultaneous presses or releases of 
a subset of fingers. When such simultaneous activity is 
detected, it signals the typing recognizer to ignore or cancel 
keystroke processing for fingers contained in the synchro- 
nous subset. It also passes on the combination of finger 
identities in the synchronous subset to the chord motion 
recognizer 18. 

The motion component extraction module 16 computes 
multiple degrees of freedom of control from individual 
finger motions during easily performable hand manipula- 
tions on the surface 2, such as hand translations, hand 
rotation about the wrist, band scaling by grasping with the 
fingers, and differential band tilting. 

The chord motion recognizer produces chord tap or 
motion events dependent upon both the synchronized finger 
subset identified by the synchronization detector 14 and on 
the direction and speed of motion extracted in 16. These 
events are then posted to the host communication interface 
20. 

The pen grip detection module 17 checks for specific 
arrangements of identified hand contacts which indicate the 
hand is configured as if gripping a pen. If such an arrange- 
ment is detected, it forwards the movements of the gripping 
fingers as inking events to the host communication interface 
20. These inking events can either lay digital ink on the host 
computer display for drawing or signature capture purposes, 
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or they can be further interpreted by handwriting recognition 
software which is well known in the art. The detailed steps 
within each of the above modules will be further described 
later. 

5 The host communication interface keeps events from both 
the typing recognizer 12 and chord motion recognizer 18 in 
a single temporally ordered queue and dispatches them to 
the host computer system 22. The method of communication 
between the interface 20 and host computer system 22 can 

10 vary widely depending on the function and processing 
power of the host computer. In a preferred embodiment, the 
communication would take place over computer cables via 
industry standard protocols such as Apple Desktop Bus, 
PS/2 keyboard and mouse protocol for PCs, or Universal 

1S Serial Bus (USB). In alternative embodiments the software 
processing of modules 10-18 would be performed within the 
host computer 22. The multi-touch surface apparatus would 
only contain enough hardware to scan the proximity sensor 
array 6, form proximity images 8, and compress and send 

20 them to the host computer over a wireless network. The host 
communication interface 20 would then play the role of 
device driver on the host computer, conveying results of the 
proximity image recognition process as input to other appli- 
cations residing on the host computer system 22. 

25 In a preferred embodiment the host computer system 
outputs to a visual display device 24 so that the hands and 
fingers 4 can manipulate graphical objects on the display 
screen. However, in alternative embodiments the host com- 
puter might output to an audio display or control a machine 

30 such as a robot. 

The term "proximity** will only be used in reference to the 
distance or pressure between a touch device such as a finger 
and the surface 2, not in reference to the distance between 
adjacent fingers. "Horizontal" and "vertical" refer to x and 

35 y directional axes within the surface plane. Proximity mea- 
surements are then interpreted as pressure in a z axis normal 
to the surface. The direction "inner** means toward the 
thumb of a given hand, and the direction "outer" means 
towards the pinky finger of a given hand. For the purposes 

40 of this description, the thumb is considered a finger unless 
otherwise noted, but it does not count as a fingertip. "Con- 
tact*' is used as a general term for a hand part when it touches 
the surface and appears in the current proximity image, and 
for the group and path data structures which represent it. 

45 FIG. 2 is a schematic diagram of a device that outputs a 
voltage 58 dependent on the proximity of a touch device 38 
to a conductive sense electrode 33. The proximity sensing 
device includes two electrical switching means 30 and 31 
connected together in series having a common node 48, an 

50 input node 46, and an output node 45. A thin dielectric 
material 32 covers the sensing electrode 33 that is electri- 
cally connected to the common node 48. A power supply 34 
providing an approximately constant voltage is connected 
between reference ground and the input node 46. The two 

55 electrical switches 30 and 31 gate the flow of charge from 
the power supply 34 to an integrating capacitor 37. The 
voltage across the integrating capacitor 37 is translated to 
another voltage 58 by a high-impedance voltage amplifier 
35. The plates of the integrating capacitor 37 can be dis- 

60 charged by closing electrical switch 36 until the voltage 
across the integrating capacitor 37 is near zero. The electri- 
cal switches 30 and 31 are opened and closed in sequence 
but are never closed at the same time, although they may be 
opened at the same time as shown in FIG. 2. Electrical 

65 switch 30 is referred to as the input switch; electrical switch 
31 is referred to as the output switch; and, electrical switch 
36 is referred to as the shorting switch. 
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The proximity sensing device shown in FIG. 2 is operated 
by closing and opening the electrical switches 30, 31, and 36 
in a particular sequence after which the voltage output from 
the amplifier 58, which is dependent on the proximity of a 
touch device 38, is recorded. Sensor operation begins with 
all switches in the open state as shown in FIG. 2. The 
shorting switch 36 is then closed for at sufficiently long time 
to reduce the charge residing on the integrating capacitor 37 
to a low level. The shorting switch 37 is then opened. The 
input switch 30 is then closed thus allowing charge to flow 
between the power supply and the common node 48 until the 
voltage across the input switch 30 becomes zero. Charge Q 
will accumulate on the sensing electrode 33 according to 

OV(e-A)/D (1) 

where V is the voltage of the power supply 34, e is the 
permittivity of the dielectric sensing electrode cover 32 and 
the air gap between the cover and the touch device 38, D is 
the thickness of this dielectric region, and A is the overlap 
area of the touch device 38 and the sensing electrode 33. 
Therefore, the amount of charge accumulating on the sens- 
ing electrode 33 will depend, among other things, on the area 
of overlap of the touch device 38 and the sensing electrode 
33 and the distance between the touch device 38 and the 
sensing electrode 33. The input switch 30 is opened after the 
voltage across it has become zero, or nearly so. Soon after 
input switch 30 is opened the output switch 31 is closed until 
the voltage across it is nearly zero. Closing the output switch 
31 allows charge to flow between the sensing electrode 33 
and the integrating capacitor 37 resulting in a voltage change 
across the integrating capacitor 37 according to: 

delta V»(V-Vc)/(l+C"D/c*A) (2) 

where Vc is the voltage across the integrating capacitor 37 
before the output switch 31 was closed, C is the capacitance 
of the integrating capacitor 37, and A and D are equal to their 
values when input switch 30 was closed as shown in 
Equation 1. Multiple switchings of the input 30 and output 
31 switches as described above produce a voltage on the 
integrating capacitor 37 that reflects the proximity of a touch 
device 38 to the sensing electrode 33. 

FIG. 3A is a schematic diagram of the proximity sensor in 
which the shorting transistor 36 and the voltage-to-voltage 
translation device 35 are replaced by a resistor 40 and a 
current-to-voltage translation device 41, respectively. The 
integrating function of capacitor 37 shown in FIG. 2 is, in 
this variation of the proximity sensor, carried out by the 
capacitor 39 shown in FIG. 3A_ Those skilled in the art will 
see that this variation of the proximity sensor produces a 
more linear output 58 from multiple switchings of the input 
and output switches, depending on the relative value of the 
resistor 40. Alternatively, the resistor 40 can be replaced by 
a shorting switch 69 (cf. FIG. 3B) to improve linearity. 
Although, the circuits shown in FIG. 3 provide a more linear 
output than the circuit shown in FIG. 2 the circuits of FIG. 
3 generally require dual power supplies while the circuit of 
FIG, 2 requires only one. 

The electrical switches shown in FIG. 2 can be imple- 
mented with various transistor technologies: discrete, 
integrated, thin film, thick film, polymer, optical, etc. One 
such implementation is shown in FIG. 4A where field effect 
transistors (FETs) are used as the input 30, output 31, and 
shorting 36 switches. The FETs are switched on and off by 
voltages applied to their gate terminals (43, 44, and 55). For 
the purpose of this description we will assume the FET is 
switched on when its gate voltage is logic 1 and switched off 
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when its gate voltage is logic 0. A controller 42 is used to 
apply gate voltages as a function of time as shown in FIG. 
4B. In this example, a sequence of three pairs of pulses (43 
and 44) are applied to the input and output transistor gates. 

5 Each pair of pulses 43 and 44 produces a voltage change 
across the integrating capacitor 37 as shown in Equation 2. 
The number of pulse pairs applied to input 43 and output 44 
gates depends on the desired voltage across integrating 
capacitor 37. In typical applications the number is between 

10 one and several hundred pulse-pairs. 

FIG. 5 shows the proximity sensor circuitry appropriate 
for use in a system comprising an array of proximity sensors 
47 as in a multi-touch surface system. The proximity sensor 
47 consists of the input transistor 30, the output transistor 31, 

15 the sensing electrode 33, the dielectric cover 32 for the 
sensing electrode 33, and conductive traces 43, 44, 45, and 
46. The conductive traces are arranged so as to allow the 
proximity sensors 47 comprising a 2D array to be closely 
packed and to share the same conductive traces, thus reduc- 

20 ing the number of wires needed in a system. FIG. 6 shows 
an example , of such a system where the input nodes 46 of all 
proximity sensors are connected together and connected to 
a power supply 34. The output nodes 45 of all proximity 
sensors are connected together and connected to a single 

25 integrating capacitor 37, a single shorting transistor 36, and 
a single voltage-to-voltage amplifier 35. In this 
implementation, a single proximity sensor 47 is enabled at 
a time by applying a logic 1 signal first to its input gate 43 
and then to its output gate 44. This gating of a single 

30 proximity sensor 47 one at a time is done by input gate 
controller 50 and output gate controller 51. For example, to 
enable the proximity sensor 47 in the lower right coner the 
input gate controller 50 would output a logic one pulse on 
conductive trace 43a. This is followed by a logic one pulse 

35 on conductive trace 44h produced by output gate controller 
51. Repetition of this pulse as shown in FIG. 4B wouldl 
cause charge to build up on integrating capacitor 37 and a 
corresponding voltage to appear at the output of the ampli- 
fier 58. The entire array of proximity sensors 47 is thus 

40 scanned by enabling a single sensor at a time and -recording 
its output. 

FIG. 7 A is a schematic of typical circuitry useful for 
converting the proximity sensor output 58 to a digital code 
appropriate for processing by computer. The proximity 

45 sensor output 58 is typically non-zero even when there is no 
touch device (e.g., ref. no. 38 in FIG. 2) nearby. This 
non-zero signal is due to parasitic or stray capacitance 
present at the common node 48 of the proximity sensor and 
is of relatively constant value. It is desirable to remove this 

50 non-zero background signal before converting the sensor 
output 58 to a digital code. This is done by using a 
differential amplifier 64 to subtract a stored record of the 
background signal 68 from the sensor output 58. The result- 
ing difference signal 65 is then converted to a digital code by 

55 an ADC (analog to digital converter) 60 producing a K-bit 
code 66. The stored background signal is first recorded by 
sampling the array of proximity sensors 47 (FIG. 6) with no 
touch devices nearby and storing a digital code specific for 
each proximity sensor 47 in a memory device 63. The 

60 particular code corresponding to the background signal of 
each proximity sensor is selected by an M-bit address input 
70 to the memory device 63 and applied 69 to a DAC (digital 
to analog converter) 61. 
The 2D array of proximity sensors 47 shown in FIG. 6 can 

65 be connected in groups so as to improve the rate at which the 
entire array is scanned. This is illustrated in FIG. 8 where the 
groups are arranged as columns of proximity sensors. In this 
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approach, the input nodes of the proximity sensors are 
connected together and connected to a power supply 34, as 
in FIG. 6. The output gates 44 are also connected in the same 
way. However, the input gates 43 are now all connected 
together and the output nodes 45 are connected to only those 
proximity sensors 47 within a row and to a dedicated voltage 
amplifier 35. With this connection method, all of the prox- 
imity sensors in a column are enabled at a time, thus 
reducing the time to scan the array by a factor N, where N 
is the number of proximity sensors in a group. The outputs 
SHa-h could connect to dedicated converter circuitry as 
shown in FIG. 7A or alternatively each output SHa-h could 
be converted one at a time using the circuitry shown in FIG. 
7B. In this figure, the output signals from each group 58a-h 
are selected one at a time by multiplexer 62 and applied to 
the positive input of the differential amplifier 64. With this 
later approach, it is assumed that the ADC 60 conversion 
time is much faster than the sensor enable time, thus 
providing the suggested speed up in sensor array scanning. 

FIG. 9 shows a typical circuit useful for the control of the 
proximity sensor's output gate 44. It consists of three input 
signals 75, 76, 78 and two output signals 44, 77. The output 
gate signal 44 is logic 1 when both inputs to AND gate 79 
are logic 1. The AND input signal 77 becomes logic 1 if 
input signal 76 is logic 1 when input signal 78 transitions 
from logic 0 to logic 1, otherwise it remains logic 0. A linear 
array of these circuits 81 can be connected end-to-end to 
enable the output gates of a single group of proximity 
sensors at a time as shown in FIG. 8. 

FIG. 10 shows a cover for the multi-touch surface 89 that 
permits the system to be sensitive to pressure exerted by 
non-conducting touch objects (e.g., gloved fingers) contact- 
ing the multi-touch surface. This cover comprises a deform - 
able dielectric touch layer 85, a deformable conducting layer 
86, and a compliant dielectric layer 87. The touch surface 85 
would have a symbol set printed on it appropriate for a 
specific application, and this surface could be removed and 
replaced with another one having a different symbol set The 
conducting layer 86 is electrically connected 88 to the 
reference ground of the proximity sensor's power supply 34. 
When a touch object presses on the top surface 85 it causes 
the conducting surface 86 under the touch device to move 
closer to the sensing electrode 33 of the proximity sensor. 
This results in a change in the amount of charge stored on 
the sensing electrode 33 and thus the presence of the touch 
object can be detected. The amount of charge stored will 
depend on the pressure exerted by the touch object More 
pressure results in more charge stored as indicated in Equa- 
tion 1. 

To obtain a softer touch surface on the multi-touch device 
a thicker and more compliant dielectric cover could be used. 
However, as the dielectric thickness increases the effect of 
the touch device on the sensing electrodes 33 spreads out 
thus lowering spatial resolution. A compliant 
anisotropically-conducting material can be used to counter 
this negative effect while also providing a soft touch surface. 
FIG. 11 shows a cover in which a compliant anisotropically- 
conducting material 90 is set between a thin dielectric cover 
85 and the sensing electrodes 33. If the conductivity of the 
compliant material 90 is oriented mostly in the vertical 
direction, the image formed by a touch device on the surface 
85 will be translated without significant spreading to the 
sensing electrodes 33, thus preserving spatial resolution 
while providing a compliant touch surface. 

FIG. 12 shows a cross section of a multi- touch surface 
that senses both the proximity and pressure of a touch 
device. The touch layer 85 is a thin dielectric that separates 
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touch devices from the sensing electrodes 33. Proximity 
sensing is relative to this surface. The electrodes 33 and 
associated switches and conductors are fabricated on a 
compliant material 89 which is attached to a rigid metal base 

5 92. The metal base 92 is electrically connected 88 to the 
reference ground of the proximity sensor's power supply 34. 
When a touch device presses on the touch surface 85 it 
causes the sensing electrodes 33 directly below to move 
closer to the rigid metal base 92. The distance moved 

10 depends on the pressure applied and thus the pressure 
exelted by a touch device can be detected as described 
before. 

To illustrate typical properties of hand contacts as they 
appear in proximity images, FIGS. 13-15 contain sample 

15 images captured by a prototype array of parallelogram- 
shaped electrodes. Shading of each electrode darkens to 
indicate heightened proximity signals as flesh gets closer to 
the surface, compresses against the surface due to hand 
pressure, and overlaps the parallelogram more completely. 

20 Note that the resolution of these images is in no way 
intended to limit the scope of the invention, since certain 
applications such as handwriting recognition will clearly 
require finer electrode arrays than indicated by the electrode 
size in these sample images. In the discussion that follows, 

25 the proximity data measured at one electrode during a 
particular scan cycle constitutes one "pixel" of the proximity 
image captured in that scan cycle. 

FIG. 13 shows a right hand flattened against the surface 
with fingers outstretched. At the far left is the oblong thumb 

30 201 which tends to point off at about 120°. The columnar 
blobs arranged in an are across the top of the image are the 
index linger 202, middle finger 203, ring finger 204 and 
pinky finger 205. Flesh from the proximal finger joint, or 
proximal phalanges 209, will appear below each fingertip if 

35 the fingers are fully extended. The inner 207 and outer 206 
palm heels cause the pair of very large contacts across the 
bottom of the image. Forepalm calluses 213 are visible at the 
center of the hand if the palm is fully flattened. This image 
shows that all the hand contacts are roughly oval -shaped, but 

40 they differ- in -pressure, size,- orientations-eccentricity and 
spacing relative to one another. This image includes all of 
the hand parts which can touch the surface from the bottom 
of one hand, but in many instances only a few of these parts 
will be touching the surface, and the fingertips may roam 

45 widely in relation to the palms as fingers are flexed and 
extended. 

FIG. 14 shows another extreme in which the hand is 
partially closed. The thumb 201 is adducted toward the 
fingertips 202-208 and the fingers are flexed so the finger- 

50 tips come down normal instead of tangential to the surface. 
The height and intensity of fingertip contacts is lessened 
somewhat because the boney tip rather than fleshy pulp pad 
is actually touching the surface, but fingertip width remains 
the same. Adjacent fingertips 202-205 and thumb 201 are so 

55 close together as to be distinguishable only by sliaht prox- 
imity valleys 210 between them. The proximal phalange 
finger joints are suspended well above the surface and do not 
appear in the image, nor do the forepalm calluses. The palm 
heels 206, 207 are somewhat shorter since only the rear of 

60 the palm can touch the surface when fingers are flexed, but 
the separation between them is unchanged. Notice that the 
proximity images are uncluttered by background objects. 
Unlike optical images, only conductive objects within a few 
millimeters of the surface show up at all. 

65 FIG. 15 is a proximity image of a right hand in a pen grip 
configuration. The thumb 201 and index fingertip 202 are 
pinched together as if they were holding a pen, but in this 
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case they are touching the surface instead. Actually the 
thumb and index finger appear the same here as in FIG. 14. 
However, the middle 203, ring 204, and pinky 205 fingers 
are curled under as if making a fist, so the knuckles from the 
top of the fingers actually touch the surface instead of the 
finger tips. The curling under of the knuckles actually places 
them behind the pinched thumb 201 and index fingertip 202 
very close to the palm heels 206, 207. The knuckles also 
appear larger than the curled fingertips of FIG. 14 but the 
same size as the flattened fingertips in FIG. 13. These 
differences in size and arrangement will be measured by the 
pen grip detector 17 to distinguish this pen grip configura- 
tion from the closed and flattened hand configurations. 

FIG. 16 represents the data flow within the contact 
tracking and identification module 10. The image segmen- 
tation process 241 takes the most recently scanned proximity 
image data 240 and segments it into groups of electrodes 242 
corresponding to the distinguishable hand parts of FIG. 13. 
The filtering and segmentation rules applied in particular 
regions of the image are partially determined by feedback of 
the estimated hand offset data 252. The image segmentation 
process 241 outputs a set of electrode group data structures 
242 which are parameterized by fitting an ellipse to the 
positions and proximity measurements of the electrodes 
within each group. 

The path tracking process 245 matches up the parameter- 
ized electrode groups 242 with the predicted continuations 
of contact path data structures 243 extracted from previous 
images. Such path tracking ensures continuity of contact 
representation across proximity images. This makes it pos- 
sible to measure the velocity of individual hand contacts and 
determine when a hand part lifts off the surface, disappear- 
ing from future images. The path tracking process 245 
updates the path positions, velocities, and contact geometry 
features from the parameters of the current groups 242 and 
passes them on to the contact identification processes 247 
and 248. For notational purposes, groups and unidentified 
paths will be referred to by data structure names of the form 
Gi and Pi respectively, where the indices i are arbitrary 
except for the null group GO and null path P0. Particular 
group and path parameters will be denoted by subscripts to 
these structure names and image scan cycles will be denoted 
by bracketed indices, so that, for example, P2Jn] represents 
the horizontal position of path P2 in the current proximity 
image, and P2jn-1] represents the position in the previous 
proximity image. The contact identification system is hier- 
archically split into a hand identification process 247 and 
within-hand finger and palm identification process 248. 
Given a hand identification for each contact, the finger and 
palm identification process 248 utilizes combinatorial opti- 
mization and fuzzy pattern recognition techniques to iden- 
tify the part of the hand causing each surface contact. 
Feedback of the estimated hand offset helps identify hand 
contacts when so few contacts appear in the image that the 
overall hand structure is not apparent. 

The hand identification process 247 utilizes a separate 
combinatorial optimization algorithm to find the assignment 
of left or right hand identity to surface contacts which results 
in the most biomechanically consistent within-hand identi- 
fications. It also receives feedback of the estimated hand and 
finger offsets 252, primarily for the purpose of temporarily 
storing the last measured band position after fingers in a 
hand lift off the surface. Then if the fingers soon touch back 
down in the same region they will more likely receive their 
previous hand identifications. 

The output of the identification processes 247 and 248 is 
the set of contact paths with non-zero hand and finger 
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indices attached. For notational purposes identified paths 
will be referred to as F0 for the unidentified or null finger, 
Fl for the thumb 201, F2 for the index finger 202, F3 for the 
middle finger 203, F4 for the ring finger 204, F5 for the 
5 pinky finger 205, F6 for the outer palm heel 206. F7 for the 
inner palm heel 207. and F8 for the forepalm calluses 208. 
To denote a particular hand identity this notation can be 
prefixed with an L for left hand or R for right hand, so that, 
for example, RF2 denotes the right index finger path. When 

10 referring to a particular hand as a whole. LH denotes the left 
hand and RH denotes the right hand. In the actual algorithms 
left hand identity is represented by a -1 and right hand by 
+1, so it is easy to reverse the handedness of measurements 
taken across the vertical axis of symmetry. 

15 It is also convenient to maintain for each hand a set of 
bitfield data registers for which each bit represents 
touchdown, continued contact, or liftoff of a particular 
finger. Bit positions within each bit field correspond to the 
hand part indices above. Such registers can quickly be tested 

20 with a bit mask to determine whether a particular subset of 
fingers has touched down. Alternatively, they can be fed into 
a lookup table to find the input events associated with a 
particular finger chord (combination of fingers). Such finger 
identity bitfields are needed primarily by the synchroniza- 

25 tion detector 14 and chord motion recognizer 18. 

The last process within the tracking and identification 
subsystem is the hand position estimator 251, which as 
described above provides biasing feedback to the identifi- 

30 cation and segmentation processes. The hand position esti- 
mator is intended to provide a conservative guess 252 of 
lateral hand position under all conditions including when the 
hand is floating above the surface without touching. In this 
case the estimate represents a best guess of where the hand 

35 will touch down again. When parts of a hand are touching 
the surface, the estimate combines the current position 
measurements of currently identified hand parts with past 
estimates which may have been made from more or less 
reliable identifications. 

40- . The simplest - but. inferior, method, of, obtaining, a hand-, 
position measurement would be to average the positions of 
all the hand's contacts regardless of identity. If hand parts 
201-207 were all touching the surface as in FIG. 13 the 
resulting centroid would be a decent estimate, lying some- 

45 where under the center of the palm since the fingers and 
palm heels typically form a ring around the center of the 
palm. However, consider when only one hand contact is 
available for the average. The estimate would assume the 
hand center is at the position of this lone contact, but if the 

50 contact is from the right thumb the hand center would 
actually be 4-8 cm to the right, or if the contact is from a 
palm heel the hand center is actually 4-6 cm higher, or if the 
lone contact is from the middle finger the hand center 
unsound actually be actually 4-6 cm lower. 

55 FIG. 17 shows the detailed steps within the hand position 
estimator 251. The steps must be repeated for each hand 
separately. In a preferred embodiment, the process utilizes 
the within-hand contact identifications (250) to compute 
(step 254) for each contact an offset between the measured 

60 contact position (Fijn] ,Fiy[n]) and the default position of 
the particular finger or palm heel (Fij^Fi^) with hand 
part identity i. The default positions preferably correspond to 
finger and palm positions when the hand is in a neutral 
posture with fingers partially closed, as when resting on 

65 home row of a keyboard. Step 255 averages the individual 
contact offsets to obtain a measured hand oflset (H^Jn], 
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is not touching the surface, Hpjn] is made small enough 

<=7 (3) that the estimated offsets gradually decay to zero at about the 

same rate as a hand lazily returns to default position. 

= — — When H^fn] is made small due to low identification 

X Fim**[n] 5 confidence, the filter tracking delay becomes large enough to 

lag behind a pair of quickly moving fingers by several 

'=7 (4) centimeters. The purpose of the filter is to react slowly to 

£Fin^*](Rj[n)-Fid&) questionable changes in contact identity, not to smooth 

Hmoy{n\ = — — contact motion. This motion tracking delay can be safely 

2 fUW 10 eliminated by adding the contact motion measured between 

images to the old oflset estimate. Step 257 obtains motion 
from the average, (H mv Jn],H mvvr [n]), of the current contact 

Preferably the weighting Fi^^n] of each finger and palm velocities: 
heel is approximately its measured total proximity, i.e. 

^^w**[ a ]**PiJ n ]* Th^ ensures that lifted fingers, whose is /=i (5) 

proximity is zero, have no influence on the average, and that X ^w^M^M 

contacts with lower than normal proximity, whose measured H mxx [n\ = /=l , 

positions and identities are less accurate, have low influence . £ ri ma M 
Furthermore, if palm heels are touching, their large total 

proximities will dominate the average. This is beneficial 20 w (6) 

because the palm heels, being immobile relative to the hand £ Ft *W Fj vW 

center compared to the highly flexible fingers, supply a more H^n] = — 

reliable indication of overall hand position. When a hand is jV^f/t] 

not touching the surface, i.e. when all proximities are zero, i=l 

the measured offsets are set to zero. This will cause the 25 

filtered hand position estimate below to decay toward the The current contact velocities, (Fi JnlFiJn]), are retrieved 

default hand position. from me pat h tracking process 245, which measures them 

As long as the contact identifications are correct, this hand independent of finger identity. Step 258 updates the esti- 

position measurement method eliminates the large errors mated hand offsets (H tf j;n],H^n])using the complete filter 

caused by assuming lone contacts originate from the center 30 equations: 
of the hand. Flexing of fingers from their default positions 

will not perturb the measured centroid more than a couple H^n^H^njH^n^i-H^JnlJCH^jQ-i^^JnlAt) (7) 
centimeters. However, this scheme is susceptible to contact 

misidentification, which can cause centroid measurement H^nl-H^nlH^Jn^ti-H^JnlJCHJn-il+H^nlAt) (8) 

errors of up to 8 cm if only one hand part is touching. 35 pinaUy, to provide a sirnilarly conservative estimate of the 

Therefore, the current measured oflfcets are not used directly, itkms of particular fingers , step 25 9 computes individual 

but are averaged with previous oflset estimates (H f Jn-1], fi ofifects (Fi c Jn],Fi ro In]) from the dbtaoce between 

H^n-1]) using a simple first-order autoregressive filter identificd COQtacts and corresponding default finger 

forming current offset -estimates O^MM M positions i ess the estimated hand offsets. For each identifi- 

S{ *r adjusts-foe-^ 40 able "contact i; the offsets aie rcmputed"**: " " — 
confidence in the current contact identifications. Since finger 

identifications accumulate reliability as more parts of the F^Qj-r^n](H n ,^ Q l-H^n>HlQ^r 1 ^(i-H^nI)(Fi^ 

hand contact the surface, one simple measure of identifica- [n-i}+FUIn]At) (9) 

don confidence is the number of fingers which have touched * 

down from the hand since the hand last left the surface. 45 Fw>)^n]^ 

... . . Ln-lJ+rt.JnJat) (10) 
Contacts with large total proximities also improve identifi- 
cation reliability because they have strong disambiguating These finger offsets reflect deviations of finger flexion and 
features such as size and orientation. Therefore HJjn] is set extension from the neutral posture. If the user places the 
roughly proportional to the maximum finger count plus the fingers in an extreme configuration such as the flattened 
sum of contact proximities for the hand. H^Jn] must of 50 hand configuration, the collective magnitudes of these finger 
course be normalized to be between zero and one or the filter offsets can be used as an indication of user hand size and 
will be unstable. Thus when confidence in contact identifi- finger length compared to the average adult 
cations is high, i.e. when many parts of the band firmly touch The parameters (H^Jn]JH eoy [n]) and (Fi eo Jn],Fi tfC Jn]) 
the surface, the autoregressive filter favors the current offset for each hand and finger constitute the estimated hand and 
measurements. However, when only one or two contacts 55 finger offset data 252, which is fed back to the segmentation 
have reappeared since hand liftoff, the filter emphasizes and identification processes during analysis of the next 
previous offset estimates in the hope that they were based proximity image. If the other processes need the estimate in 
upon more reliable identifications. absolute coordinates, they can simply add (step 260) the 
The filtered offsets must also maintain a conservative supplied offsets to the default finger positions, but in many 
estimate of hand position while the hand is floating above 60 cases the relative offset representation is actually more 
the surface for optimal segmentation and identification as convenient. 

the hand touches back down. If a hand lifts off the surface It should be clear to those skilled in the art that many 

in the middle of a complex sequence of operations and must improvements can be made to the above hand position 

quickly touch down again, it will probably touch down close estimation procedure which remain well within the scope of 

to where it lifted off. However, if the operation sequence has 65 this invention, especially in the manner of guessing the 

ended, the hand is likely to eventually return to the neutral position of lifted hands. One improvement is to make the 

posture, or default position, to rest. Therefore, while a hand estimated hand offsets decay toward zero at a constant speed 
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when a hand is lifted rather than decay exponentially. Also, 
the offset computations for each hand have been indepen- 
dent as described so far. It is actually advantageous to 
impose a minimum horizontal separation between the esti- 
mated left hand position and estimated right hand position 
such that when a hand such as the right hand slides to the 
opposite side of the board while the other hand is lifted, the 
estimated position of the other hand is displaced. In this case 
the estimated position of the lifted left hand would be forced 
from default to the far left of the surface, possibly off the 
surface completely. If the right hand is lifted and the left is 
not, an equation like the following can be applied to force 
the estimated right hand position out of the way: 

Rh^n]:=min(RtI^n],(li^l^ A -m^)+Lh^c}+miii_^nd_ 
sep) (11) 

where (LFl^-RFl^^) is the default separation between 
left and right thumbs, is the minimum horizontal separation 
to be imposed, and LH^Jn] is the current estimated offset 
of the left band. 

FIG. 18 represents the data flow within the proximity 
image segmentation process 241. Step 262 makes a spatially 
smoothed copy 263 of the current proximity image 240 by 
passing a two-dimensional diffusion operator or Gaussian 
kernel over it. Step 264 searches the smoothed image 263 for 
local maximum pixels 265 whose filtered proximity exceeds 
a significance threshold and exceeds the filtered proximities 
of nearest neighbor pixels. The smoothing reduces the 
chance that an isolated noise spike on a single electrode will 
result in a local maximum which exceeds the significance 
threshold, and consolidates local maxima to about one per 
distinguishable fleshy contact. 

Process 268 then constructs a group of electrodes or 
pixels which register significant proximity around each local 
maximum pixel by searching outward from each local 
maximum for contact edges. Each electrode encountered 
belore reaching a contact boundary is added to the local 
maximum's group. FIG. 19 shows the basic boundary elec- 
trode search pattern for an example contact boundary 274. In 
this diagram, an electrode- or image-pixel-lies at- the tip of 
each arrow. The search starts at the local maximum pixel 
276, proceeds to the left pixels 277 until the boundary 274 
is detected. The last pixel before the boundary 278 is marked 
as an edge pixel, and the search resumes to the right 279 of 
the local maximum pixel 276. Once the left and right edges 
of the local maximum's row have been found, the search 
recurses to the rows above and below, always starting 281 in 
the column of the pixel in the previous row which had the 
greatest proximity. As the example illustrates, the resulting 
set of pixels or electrodes is connected in the mathematical 
sense but need not be rectangular. This allows groups to 
closely fit the typical oval-shape of flesh contacts without 
leaving electrodes out or including those from adjacent 
contacts. 

If contacts were small and always well separated, edges 
could simply be established wherever proximity readings 
fell to the background level. But sometimes fingertips are 
only separated by a slight valley or shallow saddle point 210. 
To segment adjacent fingertips the partial minima of these 
valleys must be detected and used as group boundaries. 
Large palm heel contacts, on the other hand, may exhibit 
partial minima due to minor nonuniform! ties in flesh prox- 
imity across the contact. If all electrodes under the contact 
are to be collected in a single group, such partial minima 
must be ignored. Given a hand position estimate the seg- 
mentation system can apply strict edge detection rules in 
regions of the image where fingertips and thumb are 
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expected to appear but apply sloppy edge detection rules in 
regions of the image where palms are expected to appear. 
This ensures that adjacent fingertips aren't joined into a 
single group and that each palm heel isn't broken into 

5 multiple groups. 

Step 266 of FIG. 18 defines the positions of these seg- 
mentation regions using the hand position estimates 252 
derived from analyses of previous images. FIG. 20A shows 
the extent of the strict and sloppy segmentation regions 

10 while the hands are in their default positions, making 
estimated offsets for both hands zero. Plus signs in the 
diagram 252 indicate the estimated position of each finger 
and palm heel in each hand. Rectanigular outlines in the 
lower corners represent the left 284 and right 286 sloppy 

15 segmentation regions, where partial minima are largely 
ignored. The T-shaped region remaining is the strict seg- 
mentation region 282, where proximity saddle points must 
serve as contact boundaries. As a preferred embodiment the 
sloppy regions are rectangular, their inner boundaries 285 

20 are placed just inside of the columns where the index fingers 
202 are expected to lie, and the upper boundaries 287 are 
placed at the estimated vertical levels of their respective 
thumbs 201. The outer and lower boundaries of the sloppy 
regions are determined by the outside edges of the surface. 

25 Due to the decay in estimated hand offsets after hands leave 
the surface, the sloppy segmentation regions return to the 
positions shown after the bands have stayed off the surface 
a few seconds, regardless of hand position at liftoff. FIG. 
20B shows how the sloppy regions follow the estimated 

30 hand positions 252 as the right hand moves toward the upper 
left and the left hand moves toward the lower left. This 
ensures that the palms and only the palms fall in the sloppy 
regions as long as the hand position estimates are correct. 
FIG. 20C shows that the left sloppy region 284 is moved 

35 left off the surface entirely when the left hand is lifted off the 
surface and the right hand slides to the left side of the 
surface. This prevents the fingers of one hand from entering 
the sloppy segmentation region of the opposite hand. This 
effect is implemented by imposing a minimum horizontal 

40 separation between the sloppy -regions and, - should the - 
regions get too close to one another, letting the hand with the 
most surface contacts override the estimated position of the 
hand with fewer contacts. FIG. 21 is a detailed flow chart of 
the edge tests which are applied at each searched electrode 

45 depending on whether the electrode is in a strict or sloppy 
segmentation region. Decision diamond 290 checks whether 
the unsmoothed proximity of the electrode is greater than the 
background proximity levels. If not, the electrode is labeled 
an edge electrode in step 304 regardless of the segmentation 

50 region or search direction, and in step 305 the search returns 
to the row maximum to recurse in another direction. If the 
unsmoothed proximity is significant, further tests are applied 
to the smoothed proximity of neighboring electrodes 
depending on whether decision diamond 292 decides the 

55 search electrode is in a sloppy or strict region. 

If a strict region search is advancing horizontally within 
a row, decision diamond 306 passes to decision diamond 308 
which tests whether the electrode lies in a horizontal or 
diagonal partial minimum with respect to its nearest neigh- 

60 bor electrodes. If so, a proximity valley between adjacent 
fingers has probably been detected, the electrode is labeled 
as an edge 314 and search resumes in other directions 305. 
If not, the search continues on the next electrode in the row 
302. If a strict region search is advancing vertically to the 

65 next row, decision diamond 306 passes to decision diamond 
310 which tests whether the electrode lies in a vertical partial 
minimum with respect to the smoothed proximity of its 
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so the search continues 302 until a background level edge is G *~/_j — 
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nearest neighbor electrodes. If so, a proximity valley step 272 of the segmentation process is to extract shape, size, 
between a finger and the thumb has probably been detected, and position parameters from each electrode group. Group 
the electrode is labeled as an edge 312 and search resumes position reflects hand contact position and is necessary to 
in other directions 305. If not, the search continues into the determine finger velocity. The total group proximity, 
next row 302. If decision diamond 294 determines that a 5 eccentricity, and orientation are used by higher level mod- 
sloppy region search is advancing horizontally within a row, ules to nelp distinguish finger, palm, and thumb contacts, 
stringent horizontal minimum tests are performed to check t * , _. ^ . , 
for the crease or proximity valley between the inner and Provided G £ is the set of electrodes in group G, e, is the 
outer palm heels. To qualify, the electrode must be more than smoothed proximity of an electrode or pixel c, and e x and 
about 2 cm horizontal distance from the originating local 10 e » are the co °rdmates on the surface of the electrode center 
maximum, as checked by decision diamond 296. Also the m centimeters, to give a basic indicator of group position, 
electrode must be part of a tall valley or partial horizontal me proximity-weighted center, or centroid, is computed 
minimum which extends to the rows above and below and from positions and proximities of the group's electrodes: 
the next-nearest neighbors within the row, as checked by 

decision diamond 298. If so, the electrode is labeled as an 15 C t - £ e z (12 ^ 

edge 300 and search recurses in other directions 305. All ^ Ce 
other partial minima within the sloppy regions are ignored, 

reached on an upcoming electrode. ' 6 E 

In sloppy segmentation regions it is possible for groups to 20 „ ei e, (14) 

overlap significantly because partial minima between local G > = la ~gJ 

maxima don't act as boundaries. Typically when this hap- ** Ce 
pens the overlapping groups are part of a large fleshy contact 

such as a palm which, even after smoothing, has multiple . t , . . . „ 

local maxima. Two groups are defined to be overlapping if 25 N ° te SmCe * e i0i f. V TOX ™ lt * G * ******* 

the search originating local maximum electrode of one P«™y over each pixel in the group, it depends upon both 

group is also an element of the other group. In the interest of me S12e of a hand P art ' smce lar 8 e hand P arts tend to cause 

of presenting only one group per distinguishable fleshy groups with more pucels, and of the proximity to or pressure 

contact to the rest of the system, step 270 of FIG. 18 on ^ surface of a hand part. 

combines overlapping groups into single supergroups before 30 Since most groups are convex, their shape is well approxi- 

parameter extraction. Those skilled in the art will realize that mated by ellipse parameters. The ellipse fitting procedure 

feedback from high-level analysis of previous images can be requires a unitary transformation of the group covariance 

applied in various alternative ways to improve the segmen- matrix G C£W of second moments G^G^G : 

tation process and still lie well within the scope of this yy 
invention. For example, additional image smoothing in 35 
sloppy segmentation regions could consolidate each palm 
heel contact into a single local maximum which would pass 

strict segmentation region boundary tests. Care must be q _ V e {c x -e x f ( 16 ) 

taken with this approach however, because too much £c E 

smoothing can cause finger pairs which unexpectedly enter* 40 — - — - - — ._. 

sloppy palm regions to be joined into one group. Once a c JX = g x> = £ e t {G x -e x ){G y -e y ) ( 17 ) 

finger pair is joined, the finger identification process 248 has ** g e 
no way to tell that the fingertips are actually not a single 

palm heel, so the finger identification process will be unable Gjy ~ 2j e * (C > " € *> 

to correct the hand position estimate or adjust the sloppy 45 

regions for proper segmentation of future images. 

More detailed forms of feedback than the hand position -r, , , * r ™„„v™ f • n 

. , „. , „ _ , . . , *ri e eigenvalues Kq and A., or the covanance matrix 

estimate can be utilized as well. For example, the proximal . f _ • „- • i _*u j * o 

i_ i nnn • ^ ^ „ : ' 5 , determine the ellipse axis lengths and orientation G ft : 

phalanges (209 in FIG. 13) are actually part of the finger but 0 

tend to be segmented into separate groups than the fingertips 50 ^~ 

by the vertical minimum test 310. The vertical minimum test * ' 

is necessary to separate the thumb group from index finger- ^- 
tip group in the partially closed FIG. 14 and pen grip FIG. ""*"" 
15 hand configurations. However, the proximal phalanges of / A* - c v 

flattened fingers can be distinguished from a thumb behind 55 c g = arctaJ c J 

a curled fingertip by the fact that it is very difficult to flatten v 
one long finger without flattening the other long fingers. To 

take advantage of this constraint, a flattened finger flag 267 where Ge is uniquely wrapped into the range (0,180°). 

is set whenever two or more of the contacts identified as „ , ., , ^ . 

index through pinky in previous images are larger than 60 , For »° ve » ience whl e distinguishing fingertips from 
normal, reliably indicating that fingertips are flattening. f ^ U m 8 her s ^ em ^ *** ma J or and mu,or «"» 
Then decision diamond 310 is modified during processing of kn 8 ths "* c ° nvcrtcd ™> ^ raUo ™ t0 10 eccentricity G t : 
the current image to ignore the first vertical maximum 

encountered during search of rows below the originating c t = Gm4or (22) 

local minimum 276. This allows the proximal phalanges to 65 Gmimor 
be included in the fingertip group but prevents fingertip 
groups from merging with thumbs or forepalms. The last 



(15) 



(20) 
(21) 
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Note that since the major axis length is always greater than where the squared Euclidean distance is an easily computed 

or equal to the minor axis length, the eccentricity will always distance metric: 

be greater than or equal to one. Finally, the total group 2 22 

proximity is empirically renormalized so that the typical d ( Gk ' Pl HGVPU*)MGV p /^y) 3 ( 28 > 

curled 6ngertip will have a total proximity around one: 5 step 324 then finds for each active path PI, the closest active 



r C t (23) 



group and records the distance to it: 



Zrwprwv JWtfC = arg min d 2 (Gk, PI) V Pie PA (29) 

ci«c 

10 

On low resolution electrode arrays, the total group prox- pl ck»atc^ = <* 2 (c*, Pi) v Pie pa (30) 
imity G z is a more reliable indicator of contact size as well 
as finger pressure than the fitted ellipse parameters. 

Therefore, if proximity images have low resolution, the In step 326, an active group Gk and path PI are only paired 

orientation and eccentricity of small contacts are set to 15 with one another if they are closest to one another, i.e. 

default values rather than their measured values, and total Gk cUxestP and Plc/^^ refer to one another, and the distance 

group proximity is used as the primary measure of between them is less than the tracking radius. All of the 

contact size instead of major and minor axis lengths. following conditions must hold: 



FIG. 22 shows the steps of the path tracking process, 
which chains together those groups from successive prox- 



Okc/^^Pl (31) 



imity images which correspond to the same physical hand Pl c /««*G aGk (32) 

contact To determine where each hand part has moved since 



the last proximity image, the tracking process must decide 



(33) 



which current groups should be matched with which existing To aid in detection of repetitive taps of the same finger, it 

contact paths. As a general rule, a group and path arising 25 ma y oe useful to preserve continuity of path assignment 

from the same contact will be closer to one another than to between taps over the same location. This is accomplished 

other groups and paths. Also, biomechanical constraints on in step 334 by repeating steps 322-326 using only groups 

lateral finger velocity and acceleration limit how far a linger which were left unpaired above and paths which were 

can travel between images. Therefore a group and path deactivated within the last second or so due to finger liftoff, 

should not be matched unless they are within a distance 30 I Q step 336, any group which has still not be paired with 

known as the tracking radius of one another. Since the an active or recently deactivated path is allocated a new 

typical lateral separation between fingiers is greater than the pa th, representing touchdown of a new finger onto the 

tracking radius for reasonable image scan rates, touchdown surface. In step 344, any active path which cannot be so 

and liftoff are easily detected by the fact that touchdown paired with a group is deactivated, representing hand part 

usually causes a new group to appear outside the tracking 35 liftoff from the surface. 

radii of existing paths, and liftoff will leave an active path Step 346 incorporates the extracted parameters of each 

without a group within its tracking radius. To prevent group into its assigned path via standard filtering techniques, 

improper breaking of paths at high finger speeds, each path's The equations shown below apply simple autoregressive 

tracking radius can be made dependent on its existing filters to update the path position (Pln],PJ>]JPj;n5, veloc- 

speed and proximity.. ....... ... ...... . . . ^ ity (Pjn],P^nD, and. shape -(Pjn], Pj>])-pararneters from 

The first step 320 predicts the current locations of surface corresponding group parameters, but Kalman or finite 

contacts along existing trajectories using path positions and impulse response filters would also be appropriate, 

velocities measured from previous images. Applying previ- if a path P has just been started by group G at time step 

ous velocity to the location prediction improves the predic- n , i.e. a hand part has just touched down, its parameters are 
tion except when a finger suddenly starts or stops or changes 45 
direction. Since such high acceleration events occur less 
often than zero acceleration events, the benefits of velocity- 
based prediction outweigh the potentially bad predictions 
during finger acceleration. Letting PJn-1], P^fn-1] be the 
position of path P from time step n-1 and PJ[n-l], P^n-1] 50 
the last known velocity, the velocity-predicted path continu- 
ation is then: 

P^n)-P>-l>AtPjn-l] (24) 
P^aK^n-llMtPJd-l] (25) 

Letting the set of paths active in the previous image be PA, 
and let the set electrode groups constructed in the current 
image be G, step 322 finds for each group Gk the closest 

active path and records the distance to it: else if group G is a continuation of active path P[n-1] to time 

60 



initialized as follows: 






(34) 




(35) 




(36) 




(37) 


P«[n)"G« 


(38) 




(39) 




(40) 


PJnf-G/At 


(41) 



step n: 

Plal-OA+a-GJCP^n-l]) (42) 

P^^G^+a-GJCP^n-lD (43) 

= ™nd 2 (Gk,P[) VaeC (27) PlnJ-GA+fl-GJPin-l] (44) 



Gkci^jtP = arg min d 2 (Gk. Pt) VGkeG (26) 



Ptefti 



65 



P e [n)-G a O e+ (l-O a )Pe[n-l] (45) 
PioHSA+a-OJPjn-l] (46) 
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m 

(49) 



It is also useful to compute the magnitude P spceJ and angle 
P A> from the velocity vector (P^.PJ. Since the reliability of 
position measurements increases considerably with total 
proximity P^, the low-pass filter pole G CT is decreased for 
groups with total proximities lower than normal. Thus when 10 
signals are weak, the system relies heavily on the previously 
established path velocity, but when the finger firmly touches 
the surface causing a strong, reliable signal, the system relies 
entirely on the current group centroid measurement. 

The next process within the tracking module is contact 15 
identification. On surfaces large enough for multiple hands, 
the contacts of each hand tend to form a circular cluster, and 
the clusters -tend to remain separate because users like to 
avoid entangling the fingers of opposite hands. Because the 
arrangement of fingers within a hand cluster is independent 20 
of the location of and arrangement within the other hand's 
cluster, the contact identification system is hierarchically 
split. The band identification process 247 first decides to 
which cluster each contact belongs. Then a within-cluster 
identification process 248 analyzes for each hand the 25 
arrangement of contacts within the hand's cluster, indepen- 
dent of the other hand's cluster. Because within-cluster or 
finger identification works the same for each hand regardless 
of how many hands can fit on the surface, it will be described 
first. The description below is for identification within the 30 
right band. Mirror symmetry must be applied to some 
parameters before identifying left hand contacts. 

FIG. 23 shows the preferred embodiment of the finger 
identification process 248. For the contacts assigned to each 
hand, this embodiment attempts to match contacts to a 35 
template of hand part attractor points, each attractor point 
having an identity which corresponds to a particular finger 
or palm heel. This matching between contact paths and 
attractors should be basically one to one, but in the case that 
- some hand parts are not touching the surface, some attractors 40 
will be left unfilled, i.e., assigned to the null path or dummy 
paths. 

Step 350 initializes the locations of the attractor points to 
the approximate positions of the corresponding fingers and 
palms when the hand is in a neutral posture with fingers 45 
partially curled. Preferably these are the same default finger 
locations (Fi^^Fi^) employed in hand offset estimation. 
Setting the distances and angles between attractor points 
from a half-closed hand posture allows the matching algo- 
rithm to perform well for a wide variety of finger flexions 50 
and extensions. 

The resulting attractor points tend to lie in a ring as 
displayed by the crosses in FIG. 24. The identities of 
attractor points 371-377 correspond to the identities of hand 
parts 201-207. If the given hand is a left hand, the attractor 55 
ring must be mirrored about the vertical axis from that 
shown. FIG. 24 also includes line segments 380 forming the 
\foronoi cell around each attractor point. Every point within 
an attractor's Voronoi cell is closer to that attractor than any 
other attractor. When there is ooly one contact in the cluster 60 
and its features are not distinguishing, the assignment algo- 
rithm effectively assigns the contact to the attractor point of 
the \fcronoi cell which the contact lies within. When there 
are multiple surface contacts in a hand cluster, they could all 
lie in the same Voronoi cell, so the assignment algorithm 65 
must perform a global optimization which takes into account 
all of the contact positions at once. 



Alternative embodiments can include additional attractors 
for other hand parts or alternative, attractor arrangements for 
atypical hand configurations. For example, attractors for 
forepalm contacts can be placed at the center of the riDg, but 
since the forepalms typically don't touch the surface unless 
the rest of the hand is flattened onto the surface as well, 
forepalm attractors should be weighted such that contacts 
are assigned to them only when no regular attractors are left 
unassigned. 

For optimal matching accuracy the ring should be kept 
roughly centered on the hand cluster. Therefore step 352 
translates all of the attractor points for a given hand by the 
hand's estimated position offset. The final attractor positions 
(AjxE n ]Ajy[nD are therefore given by: 



(50) 
(53) 



Id alternative embodiments the attractor ring can also be 
rotated or scaled by estimates of hand rotation and size such 
as the estimated finger offsets, but care must be taken that 
wrong finger offset estimates and identification errors don't 
reinforce one another by severely warping the attractor ring. 

Once the attractor template is in place, step 354 constructs 
a square matrix [d»] of the distances in the surface plane 
from each active contact path Pi to each attractor point Aj. 
If there are fewer surface contacts than attractors, the null 
path P0, which has zero distance to each attractor, takes 
place of the missing contacts. Though any distance metric 
can be used, the squared Euclidean distance, 



d^AjJn^PiJnD^CAynl-Pynl) 2 



(52) 



is preferred because it specially favors assignments wherein 
the angle between any pair of contacts is close to the angle 
between the pair of attractors assigned to those contacts. 
This corresponds to the biomechanical constraint that fin- 
gertips avoid crossing over one another, especially while 
touching a surface. 

In step 356, the distances from each contact to selected 
attractors are weighted according to whether the geometrical- 
features of the given contact match those expected from the 
hand part that the attractor represents. Since the thumb and 
palm heels exhibit the most distinguishing geometrical 
features, weighting functions are computed for the thumb 
and palm heel attractors, and distances to fingertip attractors 
are unchanged. In a preferred embodiment, each weighting 
function is composed of several factor versus feature rela- 
tionships such as those plotted approximately in FIG. 25. 
Each factor is designed to take on a default value of 1 when 
its feature measurement provides no distinguishing 
information, take on larger values if the measured contact 
feature uniquely resembles the given thumb or palm hand 
part, and take on smaller values if the measured feature is 
inconsistent with the given attractor's hand part. The factor 
relationships can be variously stored and computed as 
lookup tables, piecewise linear functions, polynomials, 
trigonometric functions, rational functions, or any combi- 
nation of these. Since assignment between a contact and an 
attractor whose features match is favored as the weighted 
distance between becomes smaller, the distances are actually 
weighted (multiplied) with the reciprocals of the factor 
relationships shown. 

FIG. 25A shows the right thumb and right inner palm heel 
orientation factor versus orientation of a contact's fitted 
ellipse. Orientation of these hand parts tends to be about 
120°, whereas fingertip and outer palm heel contacts are 
usually very close to vertical (90°), and orientation of the left 
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thumb and left inner palm heel averages 6(f. The right Mathematically the optimization can then be stated as find- 
orientation factor therefore approaches a maximum at 120°. ing the permutation {jtj, . . . , 7tj} of integer hand part 
It approaches the default value of 1 at 0°, 90°, and 180° identities {1, . . . , 7} which minimizes: 
where orientation is inconclusive of identity, and reaches a 

minimum at 60°, the favored orientation of the opposite 5 7 (54) 

thumb or palm heel. The corresponding relationship for the £ 
left thumb and inner palm heel orientation factor is flipped 
about 90°. 

FIG. 25B approximately plots the thumb size factor. Since whcrc ^ the weighted distaQCC bom contact { t0 attractor 

thumb size as indicated by total proximity tends to peak at 10 ^ ^ contact i and attractor j are considered assigned to one 

two or three times the size of the typical curled fingertip, the another when jt,=j . This combinatorial optimization 

thumb size factor peaks at these sizes. Unlike palm heels, problem, known more specifically in mathematics as an 

thumb contacts can't be much larger than two or three times assignment problem, can be efficiently solved by a variety of 

the default fingertip size, so the thumb factor drops back well-known mathematical techniques, such as branch and 

down for larger sizes. Since any hand part can appear small 15 bound, localized combinatorial search, the Hungarian 

when touching the surface very lightly orjust starting to method, or network flow solvers. Those skilled in the art will 

touchdown, small size is not distinguishing, so the size recognize that this type of combinatorial optimization prob- 

factor defaults to 1 for very small contacts. km has a mathematically equivalent dual representation in 

FIG. 25C approximately plots the palm heel size factor. which the optimization is reformulated as a maximization of 

As more pressure is applied to the palms, the palm heel 20 a sum of <^ al parameters. Such reformulation of the above 

contacts can grow quite large, remaining fairly round as they Dand P art identification method as the dual of attractor- 

do so. Thus the palm heel size factor is much like the thumb contact distance minimization remains well within the scope 

size factor except the palm factor is free to increase indefi- ^if 1 * 5 invention. . ...... ^™ 

nitely. However, fingertip contacts can grow by becoming T° ™? uonec ^ computation, decision diamond 360 
taller as the fingers are flattened But shTce finger width * ^ cnds thc ^ c f r ^entification process at this stage if the hand 
4 4 . & . . - dt* jt * * assignment of the given contact cluster is only a tentative 
constant, the eccentricity of an ellipse fitted to a growmg h ^ mesis bein e * aluated b the hand i6&nX ic^ mod- 
fingertip contact increases in proportion to the height. To u f/ 24? nov/ ^ ^ the rfen hand assignments are the 
prevent flattened fingers from having a large palm factor, the final prcfcrred hypothesis, further processes verify finger 
size measure is modified to be the ratio of total contact identities and compile identity statistics such as finger 
proximity to contact eccentricity. This has little effect for 30 counts. 

palms, whose eccentricity remains near 1, but cancels the The identifications produced by this attractor assignment 

high proximities of flattened fingertips. Though directly method are highly reliable when all five fingers are touching 

using fitted ellipse width would be less accurate for low the surface or when thumb and palm features are unambigu- 

resolution electrode arrays, the above ratio basically cap- ous. Checking that the horizontal coordinates for identified 

tures contact width. 35 fingertip contacts are in increasing order easily verifies that 

Another important distinguishing feature of the palm fingertip identities are not erroneously swapped. However, 

heels is that wrist anatomy keeps the centroids of their when only two to four fingers are touching, yet no finger 

contacts separated from one other and from the ringers by strongly exhibits thumb size or orientation features, the 

several centimeters. This is not true of the thumb and assignment of the innermost finger contact may wrongly 
fingertips, which. caa be mo ved.within a. centimeter of one . ^ indicate whether the . contact is the thumb. In this case, 

another via flexible joints. The inter-palm separation feature decision diamond 362 employs a thumb verification process 

is measured by searching for the nearest neighbor contact of J 68 to measurements between the innermost 

a given contact and measuring the distance to the neighbor. fin S er olhe , r If ^wc farther mea- 

As plotted approximately in FIG. 25D, the palm separation ^ * T ^ 

c f ■ 11 j * • . _ identity is wrong, the thumb verification process changes the 

factor quickly decreases as the scpanbon .between the 45 ug ^ 9ai of & innermost finger contact. Once the finger 

contact and its nearest neighbor falls below a few assignments are verffieo; step 364 compfles statistic about 

centimeters, indicating that the given contact (and its nearest mc assignments within each hand such as the number of 

neighbor) are not palm heels. Unlike the size and orientation touching fingertips and bitfields of touching finger identities, 

factors, which only become reliable as the weight of the These statistics provide convenient summaries of identifi- 

hands fully compresses the palms, the palm separation factor 50 cat ion results for otner modules. 

is especially helpful in chstinguishing the palm heels from pjc. 26 shows the steps within the thumb verification 

pairs of adjacent fingertips because it applies equally well to module. The first 400 is to compute several velocity, 

light, small contacts. separation, and angle factors for the innermost contact 

Once the thumb and palm weightings have been applied identified as a finger relative to the other contacts identified 

to the distance matrix, step 358 finds the one-to-one assign- 55 ^ fingers. Since these inter-path measurements presuppose 

ment between attractors and contacts which mmimizes the a contact identity ordering, they could not have easily been 

sum of weighted distances between each attractor and it's included as attractor distance weightings because contact 

assigned contact. For notational purposes, let a new matrix identities are not known until the attractor distance minimi- 

[ Ci/ ] hold the weighted distances: zation ^ complete. For the factor descriptions below, let FI 

60 be the innermost finger contact, FN be the next innermost 

KPitkMmb^jbct ^irwwjtw) if J ~ l (53) finger contact, and FO be the outermost finger contact. 

4* if 2 & 5 The separation between thumb and index finger is often 

dijHPip^^joa Pip^.svj* if ; = 6 ^rger than the separations between fingertips, but all sepa- 

j 1 1 n* D . D \ 'c ' i rations tend to grow as the fingers are outstretched. There- 
65 fore an inner separation factor inner_separation_fact is 
defined as the ratio of the distance between the innermost 
and next innermost finger contacts to the average of the 
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distances between other adjacent fingertip contacts, avg_ The rotation factor must also be very selective. If the 

separation: rotation factor was simply proportional to changes in the 

angle between innermost and outermost finger, it would 

(55) erroneously grow in response to asymmetries in finger 

5 motion such as when the innermost finger starts translating 
downward while the outermost contact is stationary. To be 
more selective, the rotation factor must favor symmetric 
rotation about an imaginary pivot between the thumb and 

The factor is clipped to be greater than one since an fingertips. The approximate rotation factor equation below 

innermost separation less than the average can occur regard- 1Q peaks as the innermost and outermost finger move in oppo- 

less of whether thumb or index finger is the innermost site directions, but in this case the contacts should move 

touching finger. In case there are only two finger contacts, a perpendicular to the vector between them: 

default average separation of 2-3 cm is used. The factor rotationfact— 

tends to become larger than one if the innermost contact is vTt,^n>F^^n]xsin(Fi w Jn]-^(n[a]^0[Ql))xsinCFO^ 

actually the thumb but remains near one if the innermost Kn]-^(FiInlPO(n])) (59) 

contact is a fingertip. rotatioiL_£ict:-mAx(0, rotation_fect) (60) 

Since the thumb rarely moves further forward than the Since motions which maximize this rotation factor are 

fingertips except when the fingers are curled into a fist, the easy to perform between the opposable thumb and another 

angle between the innermost and next innermost finger finger but difficult to perform between two fingertips, the 

contact can help indicate whether the innermost finger 2Q rotation factor is a robust indicator of thumb presence, 

contact is the thumb. For the right hand the angle of the Finally, a fuzzy logic expression (step 402) combines 

vector from the thumb to the index finger is most often 60°, these inter-contact factors with the thumb feature factors for 

though it ranges to 0° as the thumb moves forward and to the innermost and next innermost finger contacts. In a 

120° as the thumb adducts under the palm. This is reflected preferred embodiment, this fuzzy logic expression for the 

in the approximate plot of the inner angle factor in FIG. 32, 2S combined_thumb_fact takes the form: 

which peaks at 60° and approaches 0 toward 0° and 120°. If combincd^umb_^ct-(inncr_separatioiL_fectxangle_fect+ex- 

the innermost finger contact is actually an index fingertip, P aMi °^ &ct+rote ^ on ^ct)x(Fi^^^ rt w e ^ lWbc( ))< 

the measured angle between innermost and next innermost ^^^i/FN^jtosjte) (61) 

contact will likely be between 30° and minus 60°, producing The feature factor ratios of this expression attempt to 

a very small angle factor. compare the features of the innermost contact to current 

The inner separation and angle factors are highly dis- 30 features of the next innermost contact, which is already 

criminating of neutral thumb postures, but users often known to be a fingertip. If the innermost contact is also a 

exceed the above cited separation and angle ranges when fingertip its features should be similar to the next innermost, 

performing hand scaling or rotation gestures. For instance, <T ausin S ^ ratios t0 rcmain near on 5; However, thumb-like 

during an anti-pinch gesture, the thumb may start pinched „ eature i, on ' he ^jmost contact will cause the ratios to be 

against the index or middle fingertip, but then the thumb and 35 S^L 1 ?? 1 ^ 016 *T u Umb factor t e ? ceeds a 

- & c *u tl- • threshold, diamond 404 decides the innermost finger contact 

fingertip slide away from one another. This causes the inner fa a mumb lf dccisiQn diamond 4n § etermines 

separation factor to be relatively small at the start of the me conUlc ( ^ not ^ assi ^ t0 me mumb attractor 

gesture. Similarly, the thumb-index angle can also exceed 412 step 414 me mnULa assignment mward on the 

the range expected byjhe inner angle factor at the beginning ^ attractor ring to the thumb attractor. Otherwise, if decision * 

or end of hand rotation gestures, wherein the fingers route diamond 406 determines that the combined thumb factor is 

as if turning a screw. To compensate the inner separation and less than a low threshold, the innermost contact is most 

angle factors are fuzzy OR'ed with expansion and rotation definitely not the thumb. Therefore if decision diamond 408 

factors which are selective for symmetric finger scalings or finds the contact assigned to the thumb attractor, step 410 

rotations centered on a point between the thumb and fin- 4S shifts the innermost contact assignment and any adjacent 

geru'ps. finger contacts outward on the attractor ring to unfiU the 

When defined by the following approximate equation, the thumb attractor. If the combined__thumb_fact is between 

expansion factor peaks as the innermost and outermost the high and low threshold or if the existing assignments 

finger contacts slide at approximately the same speed and in agree with the threshold decisions, step 413 makes no 

opposite directions, parallel to the vector between them: cn assignment changes. 

50 The hand contact features and interrelationships intro- 

expa nsionfact— duced here to aid identification can be measured and com- 

vTTa^n^Fr^^n^cosCn^ bined in various alternative ways yet remain well within the 

(TO^a}-£(Fifry?0[riI)) (56) scope of the invention. In alternative embodiments of the 

multi-touch surface apparatus which include raised, touch- 

55 insensitive palm rests, palm identification and its requisite 

where Z.(F(nl FO[nJ is the angle between the fingers: attractors and factors may be eliminated. Geometrical 

parameters can be optimally adapted to measurements of 
individual user hand size taken while the hand is flattened. 



expansion_fact:~max(p, expansion_fact) " (57) 



UFi[n], FO[rib = arctan^ F ^ _ ^ ^ j However, the attractor-based identification method already 

60 tolerates variations in a single person's finger positions due 
to finger flexion and extension which are as great or greater 
Translational motions of both fingers in the same direction than the variations in hand size across adult persons. There- 
produce negative factor values which are clipped to zero by fore adaptation of the thumb and palm size factors to a 
the max operation. Computing the geometric rather than person's average finger and palm heel proximities is more 
arithmetic mean of the innermost and outermost speeds aids 65 important than adaptation of attractor positions to individual 
selectivity by producing a large expansion factor only when finger lengths, which will only add marginal performance 
speeds of both contacts are high. improvements. 
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As another example of an alternative method for incor- depend heavily on which side of the board the cluster starts 

porating these features and relationships into a hand contact on, but the identity of contacts which recently lifted off 

identifier. FIG. 27 diagrams an alternative finger identifies- nearby also proves helpful. For example, if the right hand 

tion embodiment which does not include an attractor tem- moves from the right side to the middle of the surface and 

plate. To order the paths from finger and palm contacts 5 lifts off, the next contacts which appear in the middle will 

within a given hand 430, step 432 constructs a two- most likely be from the right hand touching back down, not 

dimensional matrix of the distances from each contact to the from the left hand moving to the middle and displacing the 

other contacts. In step 434, a shortest path algorithm well right hand. The division between left and right halves of the 

known from the theory of network flow optimization then surface should therefore be dynamic, shifting toward the 

finds the shortest graph cycle connecting all the contact 10 right or left according to which hand was most recently near 

paths and passing through each once 434. Since hand the middle. Since the hand offset estimates temporarily 

contacts tend to lie in a ring this shortest graph cycle will retain the last known hand positions after liftoff, such a 

tend to connect adjacent contacts, thus establishing a sen- dynamic division is implemented by tying the positions of 

sible ordering for them. left hand and right hand attractor templates to the estimated 

The next step 438 is to pick a contact at an extreme 15 hand positions, 

position in the ring such as the innermost or outermost and Though cases remain in which the user can fool the hand 

test whether it is a thumb (decision diamond 440) or palm identification system with sudden placements of a hand in 

(decision diamond 442). This can be done using contact unexpected locations, the user may actually wish to fool the 

features and fuzzy logic expressions analogous to those system in these cases. For example, users with only one hand 

utilized in the thumb verification process and the attractor 20 free to use the surface may intentionally place the hand far 

weightings. If the innermost path is a thumb, step 444 onto the opposite half of the surface to access the chord input 

concludes that contacts above are most likely fingertips, and operations of the opposite hand. Therefore, when a hand 

contacts in the ring below the thumb are most likely palms. cluster suddenly touches down well into the opposite half of 

If (442) the innermost path is a palm heel, step 446 con- the surface, it can safely be given the opposite hairs identity, 

eludes the paths significantly above the innermost must be 25 regardless of its true identity. Arching the surface across the 

fingers while paths at the same vertical level should be middle can also discourage users from sliding a hand to the 

palms. The thumb and palm tests are then repeated for the opposite side by causing awkward forearm pronation should 

contacts adjacent in the ring to the innermost until any other users do so. 

thumb or palm contacts are found. Once any thumb and FIG. 29 shows process details within the hand idenufi- 

palm contacts are identified, step 448 identifies remaining 30 cation module 247. Decision diamond 450 first determines 

fingertip contacts by their respective ordering in the ring and whether the hand identification algorithm actually needs to 

their relatively high vertical position. be executed by checking whether all path proximities have 

Since this alternative algorithm does not include an attrac- stabilized. To maximize stability of the identifications, hand 

tor template to impose constraints on relative positions, the and finger identities need only be reevaluated when a new 

fuzzy verification functions for each contact may need to 35 hand part touches down or disambiguating features of exist - 

include measurements of the vertical position of the contact ing contacts become stronger. The contact size and orienta- 

relative to other contacts in the ring and relative to the tion features are unreliable until the flesh frilly compresses 

estimated hand offset. The attractor template embodiment is against the surface a few dozen milliseconds after initial 

preferred over this alternative embodiment because the surface contact. Therefore decision diamond 450 executes 

- attractor embodiment more elegantly incorporates expected 40 the hand identification algorithm for each proximity image 

angles between contacts and the estimated hand offset into in which a new contact appears and for subsequent prox- 

the finger identification process. imity images in which the total proximity of any new 

Hand identification is needed for multi-touch surfaces contacts continues to increase. For images in which prox- 

which are large enough to accommodate both hands simul- imities of existing contacts have stabilized and no new 

taneously and which have the left and right halves of the 45 contacts appear, path continuation as performed by the path 

surface joined such that a band can roam freely across the tracking process 245 is sufficient to retain and extend (step 

middle to either half of the surface. The simplest method of 452) the contact identifications computed from previous 

hand identification would be to assign hand identity to each images. 

contact according to whether the contact initially touched Should the hand identification algorithm be invoked for 

down in the left or right half of the surface. However, if a 50 the current image, the first step 453 is to define and position 

hand touched down in the middle, straddling the left and left and right hand attractor templates. These should be 

right halves some of the hand's contacts would end up basically the same as the attractor templates (FIG. 24, step 

assigned to the left hand and others to the right hand. 352) used in within-hand identification, except that both left 

Therefore more sophisticated methods which take into and right rings must now be utilized at once. The default 

account the clustering properties of hand contacts must be 55 placement of the rings relative to one another should cor- 

applied to ensure all contacts from the same hand get the respond to the default left and right hand contact positions 

same identity. Once all surface contacts are initially shown in FIG. 20A. Each ring translates to follow the 

identified, the path tracking module can reliably retain estimated position of its hand, just like the sloppy segmen- 

existing identifications as a hand slides from one side of the tation regions follow the hands in FIG. 20B. Individual 

surface to the other. 60 attractor points can safely be translated by their correspond- 

The thumb and inner palm contact orientations and the ing estimated finger offsets. Therefore the final attractor 

relative thumb placement are the only contact features positions (AjJnlAjyM) for Ihe left hand L and right hand 

independent of cluster position which distinguish a lone H attractor rings are: 
cluster of right hand contacts from a cluster of left hand 

contacts. If the thumb is lifted off the surface, a right hand 65 uj r [n>-Lh«„ln}fLFj e(1( [n>Lfj deA (62) 

contact cluster appears nearly indistinguishable from a left Uj^n}-Lh Jn}*LFj - Jii)»'US - ^ (63) 

hand cluster. In this case cluster identification must still ' 
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Raiio^Wnp-RFjcoi^RW ( 64 ) tor distances. Finally, the weighted left and right hand costs 
R.tf.J-»W^J»>^ (65) are and scaled by the reciprocal of a hand 
-T j«m w separauon factor to obtain a total cost for the partition. 
Basically the hand identification algorithm will compare If decision diamond 464 determines this total cost is lower 
the cost of assigning contacts to attractors in one ring versus 5 than the total costs of the partitions evaluated so far 464, step 
the other, the cost depending on the sum of weighted 466 records the partition cost as the lowest and records the 
distances between each contact and its assigned attractor. dividing contour. Decision diamond 472 repeats this process 
Adjusting the attractor ring with the estimated hand and for each contour 470 until the costs of all partitions have 
finger offsets lowers the relative costs for assignment been evaluated. Step 473 chooses the partition which has the 
hypotheses which resemble recent hand assignments, help- 10 lowest cost overall as the actual hand partitioning 473, and 
ing to stabilize identifications across successive proximity the hand identities of all contact paths are updated accord- 
images even when hands temporarily lift off. ingly. Then step 474 reinvokes the within-hand finger con- 
Next a set of assignment hypotheses must be generated tact identification process so that the thumb verification and 
and compared. The most efficient way to generate sensible statistics gathering processes are performed using the actual 
hypotheses is to define a set of roughly vertical contour 15 hand assignments- 
lines, one between each horizontally adjacent contact. Step Users often perform clutching motions in which the right 
454 does this by ordering all surface contacts by their hand, for example, lifts off from a slide at the right side of 
horizontal coordinates and establishing a vertical contour the surface, touches back down in the middle of the surface, 
halfway between each pair of adjacent horizontal coordi- and resumes sliding toward the right. Therefore when a hand 
nates. FIGS. 30A-C show examples of three different con- 20 is detected touching down in the middle of the surface and 
tours 475 and their associated assignment hypotheses for a sliding toward one side, it probably came from that side. A 
fixed set of contacts. Each contour corresponds to a separate hand velocity factor, plotted approximately in FIG. 31 A, 
hypothesis, known also as a partition, in which all contacts captures this phenomenon by slightly increasing in value 
to the left 476 of the contour are from the left hand, and all when a hand cluster's contacts are moving toward the 
contacts to the right 477 of the contour are from the right 25 cluster's assigned side of the board, thus decreasing the 
hand. Contours are also necessary at the left and right ends basic cost of the hand. The factor is a function of the average 
of the surface to handle the hypotheses that all contacts on of the contacts' horizontal velocities and the side of the 
the surface are from the same hand. Contours which hypoth- surface the given cluster is assigned. Since high speeds do 
esize more contacts on a given hand than can be caused by not necessarily give a stronger indication of user intent, the 
a single hand are immediately eliminated. 30 factor saturates at moderate speeds. 

Generating partitions via vertical contours avoids all Though the thumb orientation tactors help identify which 

hypotheses in which contacts of one hand horizontally hand a thumb is from when the thumb lies in the ambiguous 

overlap or cross over contacts of the opposite hand. Con- middle region of the surface, the vertical position of the 

sidering that each band can cause seven or more distinct thumb relative to other fingers in the same hand also gives 

contacts, this reduces the number of hand identity permu- 35 a strong indication of handedness. The thumb tends to be 

tations to examine from thousands to at most a dozen. With positioned much lower than the fingertips, but the pinky 

fewer hypotheses to examine, the evaluation of each parti- tends to be only slightly lower than the other fingertips. The 

tion can be much more sophisticated, and if necessary, handedness factor, plotted approximately in FIG. 31B, takes 

computationally costly. advantage of this constraint by boosting the hand cost when 

The optimization search loop follows. Its goal -is to 40 the contact identified as-the outermost fingertip is more than 

determine which of the contours divides the contacts into a a couple centimeters lower than the next outermost fingertip 

partition of two contact clusters such that the cluster posi- contact. In such cases the tentative hand assignment for all 

tions and arrangement of contacts within each cluster best contacts in the cluster is probably wrong. Since this causes 

satisfy known anatomical and biomechanical constraints. the within-hand identification algorithm to fit the contacts to 

Hie optimization begins by picking (step 456) a first contour 45 the wrong attractor ring, finger identities become reversed 

divider such as the leftmost and tentatively assigning (step such that the supposedly lowered pinky is truly a lowered 

458) any contacts to the left of the contour to the left hand thumb of the opposite hand. Unfortunately, limited confi- 

and the rest to the right hand. Step 460 invokes the finger dencc can be placed in the handedness factor. Though the 

identification algorithm of FIG. 23, which attempts to assign pinky should not appear lowered as much as the thumb, the 

finger and palm identities to contacts within each hand. 50 outer palm heel can, creating an ambiguity in which the 

Decision diamond 360 avoids the computational expense of thumb and fingertips of one hand have the same contact 

thumb verification 368 and statistics gathering 364 for this arrangement as the fingertips and outer palm heel of the 

tentative assignment hypothesis. opposite hand. This ambiguity can cause- the handedness 

Returning to FIG. 29, step 462 computes a cost for the factor to be erroneously low for an accurately identified 

partition. This cost is meant to evaluate how well the 55 hand cluster, so the handedness factor is only used on 

tentatively identified contacts fit their assigned attractor ring clusters in the middle of the surface where hand position is 

and how well the partition meets between-hand separation ambiguous. 

constraints. This is done by computing for each band the Distinguishing contact clusters is challenging because a 

sum of weighted distances from each tentatively identified cluster can become quite sparse and large when the fingers 

contact to its assigned attractor point as in Equation 54 of 60 outstretched, with the pinky and thumb of the same hand 

finger identification, including size and orientation feature spanning up to 20 cm. However, the palm can stretch very 

factors for thumb and palm attractors. This sum represents little in comparison, placing useful constraints on how far 

the basic template fitting cost for a hand. Each hand cost is apart palm heel contacts and fo rep alms from the same hand 

then weighted as a whole with the reciprocals of its clutching can be. The entire palm region of an outstretched adult hand 

velocity, handedness, and palm cohesion factors. These 65 is about 10 cm square, so palm contact centra ids should not 

factors, to be described below, represent additional con- be scattered over a region larger than about 8 cm. When a 

straints which are underemphasized by the weighted attrac- partition wrongly includes fingers from the opposite hand in 
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a cluster, the within-cluster identification algorithm tends to 
assign the extra fingers from the opposite hand to palm heel 
and forepalm attractors. This usually causes the contacts 
assigned to the cluster's palm attractors to be scattered 
across the surface wider than is plausible for true palm 
contacts from a single hand. To punish such partitions, the 
palm cohesion factor quickly drops below one for a tentative 
hand cluster in which the supposed palm contacts are 
scattered over a region larger than 8 cm. Therefore its 
reciprocal will greatly increase the hand's basic cost. FIG. 
31C shows the value of the palm cohesion factor versus 
horizontal separation between palm contacts. The horizontal 
spread can be efficiently measured by rinding the maximum 
and minimum horizontal coordinates of all contacts identi- 
fied as palm heels or fore palms and taking the difference 
between the maximum and minimum. The measurement and 
factor value lookup are repeated for the vertical separation, 
and the horizontal and vertical factors are multiplicatively 
combined to obtain the final palm cohesion factor. 

FIG. 33 is an approximate plot of the inter-hand separa- 
tion factor. This factor increases the total costs of partitions 
in which the estimated or actual horizontal positions of the 
thumbs from each hand approach or overlap. It is measured 
by finding the minimum of the horizontal offsets of right 
hand contacts with respect to their corresponding default 
finger positions. Similarly the maximum of the horizontal 
offsets of the left hand contacts with respect to their corre- 
sponding default finger positions is found. If the difference 
between these hand offset extremes is small enough to 
suggest the thumbs are overlapping the same columnar 
region of the surface while either touching the surface or 
floating above it, the separation factor becomes very small. 
Such overlap corresponds to a negative thumb separation in 
the plot. To encourage assignment of contacts which are 
within a couple centimeters of one another to the same 
cluster, the separation factor gradually begins to drop start- 
ing with positive separations of a few centimeters or less. 
The inter-hand separation factor is not applicable to parti- 
tions in which all surface contacts are assigned to the same 
hand* and takes on the default value of one in this case. 

Alternative embodiments of this hand identification pro- 
cess can include additional constraint factors and remain 
well within the scope of this invention. For example, a 
velocity coherence factor could be computed to favor par- 
titions in which all fingers within a cluster slide at approxi- 
mately the same speed and direction, though each cluster as 
a whole has a different average speed and direction. 

Sometimes irreversible decisions made by the chord 
motion recognizer or typing recognizer on the basis of 
existing hand identifications prevent late changes in the 
identifications of hand contacts even when new proximity 
image information suggests existing identifications are 
wrong. This might be the case for a chord slide which 
generates input events that can't be undone, yet well into the 
slide new image information indicates some fingers in the 
chord should have been attributed to the opposite hand. In 
this case the user can be warned to stop the slide aind check 
for possible input errors, but in the meantime it is best to 
retain the existing identifications, even if wrong, rather than 
switch to correct assignments which could have further 
unpredictable effects when added to the erroneous input 
events. Therefore once a chord slide has generated input 
events, the identifications of their existing paths may be 
locked so the hand identification algorithm can only swap 
identifications of subsequent new contacts. 

This hand identification process can be modified for 
differently configured multi-touch surfaces and remain well 
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within the scope of this invention. For surfaces which are so 
narrow that thumbs invade one another's space or so tall that 
one hand can lie above another, the contours need not be 
straight vertical lines. Additional contours could weave 

5 around candidate overlapping thumbs, or they could be 
perpendicular to the vector between the estimated hand 
positions. If the surface was large enough for more than one 
user, additional attractor rings would have to be provided for 
each additional hand, and multiple partitioning contours 

10 would be necessary per hypothesis to partition the surface 
into more than two portions. On a surface large enough for 
only one hand it might still be necessary to determine which 
hand was touching the surface. Then instead of hypothesiz- 
ing different contours, the hand identification module would 

15 evaluate the hypotheses that either the left hand attractor 
ring or the right hand attractor ring was centered on the 
surface. If the surface was mounted on a pedestal to allow 
access from all sides, the hand identification module would 
also hypothesize various rotations of each attractor ring. 

20 The attractor-based finger identification system 248 will 
successfully identify the individual hand contacts which 
comprise the pen grip hand configuration (FIG. 15). 
However, additional steps are needed to distinguish the 
unique finger arrangement within the pen grip from the 

25 normal arrangement within the cbsed hand configuration 
(FIG. 14). In this pen grip arrangement the outer fingers curl 
under toward the palms so their knuckles touch the surface 
and the index finger juts out ahead of them. The pen grip 
detection module 17 employs a fuzzy pattern recognition 

30 process similar to the thumb verification process to detect 
this unique arrangement. 

An additional problem with handwriting recognition via 
the pen grip hand configuration is that the inner gripping 
fingers and sometimes the whole hand will be picked up 

35 between strokes, causing the distinguishing finger arrange- 
ment to temporarily disappear. Therefore the pen grip rec- 
ognition process must have hysteresis to stay in handwriting 
mode between gripping finger lifts. In the preferred 
embodiment, hysteresis is obtained by temporal filtering of 

40 the combined- fuzzy decision - factors -and- by using -the - 
estimated finger positions in measurements of finger 
arrangement while the actual fingers are lifted off the sur- 
face. The estimated finger positions provide effective hys- 
teresis because they temporarily retain the unique jutting 

45 arrangement before decaying back toward the normal arched 
fingertip positions a few seconds after liftoff. 

FIG. 28 shows the steps within the pen grip detection 
module 17. Decision diamond 485 determines whether all 
pen grip hand parts are touching the surface. If not, decision 

50 diamond 486 causes the estimated finger and palm positions 
to be retrieved for any lifted parts in step 487 only if pen grip 
or handwriting mode is already active. Otherwise the pro- 
cess exits for lack of enough surface contacts. Thus the 
estimated finger positions cannot be used to start handwrit- 

55 ing mode, but they can continue it. Step 488 retrieves the 
measured positions and sizes of fingers and palm heels 
which are touching the surface. 

Step 489 computes a knuckle factor from the outer finger 
sizes and their vertical distance from the palm heels which 

60 peaks as the outer finger contacts become larger than normal 
fingertips and close to the palm heels. Step 490 computes a 
jutting factor from the difference between the vertical coor- 
dinates of the inner and outer fingers which peaks as the 
index fingertip juts further out in front of the knuckles. Step 

65 491 combines the knuckle and jutting factors in a fuzzy logic 
expression and averages the result with previous results via 
an autoregressive or moving average filter. Decision dia- 
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mood 492 continues or starts pen grip mode if the filtered 
expression result is above a threshold which may itself be 
variable to provide additional hysteresis. While in pen grip 
mode, typing 12 and chord motion recognition 18 are 
disabled for the pen gripping hand. 

In pen grip mode, decision diamond 493 determines 
whether the inner gripping fingers are actually touching the 
surface. If so, step 495 generates inking events from the path 
parameters of the inner fingers and appends them to the 
outgoing event queue of the host communication interface. 
These inking events can either cause "digital ink" to be layed 
on the display 24 for drawing or signature capture purposes, 
or they can be intercepted by a handwriting recognition 
system and interpreted as gestures or language symbols. 
Handwriting recognition systems are well known in the art. 

If the inner fingers are lifted, step 494 sends stylus raised 
events to the host communication interface to instruct the 
handwriting recognition system of a break between symbols. 
In some applications the user may need to indicate where the 
"digital ink" or interpreted symbols are to be inserted on the 
display by positioning a cursor. Though on a multi-touch 
surface a user could move the cursor by leaving the pen grip 
configuration and sliding a finger chord, it is preferable to 
allow cursor positioning without leaving the pen grip con- 
figuration. This can be supported by generating cursor 
positioning events from slides of the palm heels and outer 
knuckles. Since normal writing motions will also include 
slides of the palm heels and outer knuckles, palm motions 
should be ignored until the inner fingers have been lifted for 
a few hundred milliseconds. 

Should the user actually pick up a conductive stylus and 
attempt to write with it, the hand configuration will change 
slightly because the inner gripping fingers will be directing 
the stylus from above the surface rather than touching the 
surface during strokes. Since the forearm tends to supinate 
more when actually holding a stylus, the inner palm heel 
may also stay off the surface while the hand rests on the sides 
of the pinky, ring finger and the outer palm heel. Though the 
outer palm heel may lie further outward than normal with 
respect to the pinky, the ring and -pinky fingers will still 
appear as large knuckle contacts curled close to the outer 
palm. The tip of the stylus essentially takes the place of the 
index fingertip for identification purposes, remaining at or 
above the vertical level of the knuckles. Thus the pen grip 
detector can function in essentially the same way when the 
user writes with a stylus, except that the index fingertip path 
sent to the host communication interface will in actuality be 
caused by the stylus. 

Technically, each hand has 24 degrees of freedom of 
movement in all finger joints combined, but as a practical 
matter, tendon linkage limitations make it difficult to move 
all of the joints independendy. Measurements of finger 
contacts on a surface yield ten degrees of freedom in motion 
lateral to the surface, five degrees of freedom in individual 
fingertip pressure or proximity to the surface, and one degree 
of freedom of thumb orientation. However, many of these 
degrees of freedom have limited ranges and would require 
unreasonable twisting and dexterity from the average user to 
access independently. 

The purpose of the motion component extraction module 
16 is to extract from the 16 observable degrees of freedom 
enough degrees of freedom for common graphical manipu- 
lation tasks in two and three dimensions. In two dimensions 
the most common tasks are horizontal and vertical panning 
rotating, and zooming or resizing, in three dimensions, two 
additional rotational degrees of freedom are available 
around the horizontal and vertical axes. The motion com- 
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ponent extractor attempts to extract these 4-6 degrees of 
freedom from those basic hand motions which can be 
performed easily and at the same time without interfering 
with one another. When multiple degrees of freedom can be 
5 accessed at the same time they are said to be integral rather 
than separable, and integral input devices are usually faster 
because they allow diagonal motions rather than restricting 
motions to be along a single axis or degree of freedom at one 
time. 

10 When only four degrees of freedom are needed, the basic 
motions can be whole hand translation, hand scaling by 
uniformly flexing or extending the fingers, and hand rotation 
either about the wrist as when unscrewing a jar lid or 
between the fingers as when unscrewing a nut. Not only are 

15 these hand motions easy to perform because they utilize 
motions which intuitively include the opposable thumb, they 
correspond cognitively to the graphical manipulation tasks 
of object rotation and sizing. Their only drawback is that the 
translational motions of all the fingers during these hand 

20 rotations and sea lings do not cancel perfectly and can 
instead add up to a net translation in some direction in 
addition to the desired rotation or scaling. To allow all 
motions to be performed simultaneously so that the degrees 
of freedom are integral yet to prevent unintended transla- 

25 tions from imperfectly performed scalings and rotations, the 
motion extractor preferentially weights the fingers whose 
translations cancel best and nonlinearly scales velocity com- 
ponents depending on their speeds relative to one another. 
The processes within the motion component extractor 16 

30 are shown in FIG. 34. Step 500 first fetches the identified 
contact paths 250 for the given hand. These paths contain the 
lateral velocities and proximities to be used in the motion 
calculations, and the identifications are needed so that 
motion of certain fingers or palm heels which would degrade 

35 particular motion component calculations can be 
de-emphasized. 

The next step 502 applies additional filtering to the lateral 
contact velocities when finger proximity is changing rapidly. 
This is necessary because during finger liftoff and touch 

40 down on the- surface, the front part of- the -fingertip oftenr 
touches down before and lifts off after the back of the 
fingertip, causing a net downward or upward lateral trans- 
lation in the finger centroid. Such proximity -dependent 
translations can be put to good use when slowly rolling the 

45 fingertip for fine positioning control, but they can also annoy 
the user if they cause the cursor to jump away from a 
selected position during finger liftoff. This is prevented by 
temporarily down scaling a finger's lateral velocity in pro- 
portion to large changes in the finger's proximity. Since 

50 other fingers within a hand tend to shift slightly as one finger 
lifts off additional down scaling of each finger velocity is 
done in response to the maximum percent change in prox- 
imity among contacting fingers. Alternatively, more precise 
suppression can be obtained by subtracting from the lateral 

55 finger speed an amount proportional to the instantaneous 
change in finger contact height This assumes that the 
perturbation in lateral ringer velocity caused by finger liftoff 
is proportional to the change in contact height due to the 
back of the fingertip lifting off first or touching down last. 

60 Process 504, whose detailed steps are shown in FIG. 36, 
measures the polar velocity components from radial 
(scaling) and rotational motion. Unless rotation is extracted 
from thumb orientation changes, at least two contacting 
fingers are necessary to compute a radial or angular velocity 

65 of the hand. Since thumb motion is much more independent 
of the other fingers than they are of one another, scalings and 
rotations arc easier for the user to perform if one of these 



02/18/2004, EAST Version: 1.4.1 



US 6,323,846 Bl 
43 44 

fingers is the opposable thumb, but the measurement method the thumb verification process are only sensitive to sym- 

will work without the thumb. If decision diamond 522 metric rotation and scaling about a fixed point between the 

determines that less than two fingers are touching the fingers. They approach zero if any significant whole hand 

surface, step 524 sets the radial and rotational velocities of translation is occurring or the finger motions aren't comple- 

the hand to zero. FIG. 35 shows trajectories of each finger 5 me ntary. In case the user fails to properly move the outer- 

during a contractive hand scaling. The thumb 201 and pinky most during a rotation or scaling gesture, step 531 

205 travel in nearly opposite directions at roughly the same uses equations of ^ approximate form of Equations 56-60 

speed, so that the sum of their motions cancels for zero net to ute tQ{&{im ^ velocities ^tween the 

trar*lahon,bu^ mumb and ^ fi ^ ± oulermost 

for a large net scaling. The central fingers 202-204 also 10 « u - , ... f . , . • * tl _ iL 

move toward a central point but the {dm heels remain ^ jetties are preferably combined with the 

stationary, failing to complement the flexing of the central " sults of EciuaUoDS 6 ^ 8 ™ a maxiI °um operation rather 

fingers. Therefore the difference between motion of a central maa an avcra S c m casc translational motion causes the fixed 

finger and any other finger is usually less than the difference P omt nations or scalings to be zero. Finally, decision 

between the pinky and thumb motions, and the sum of 15 diam0Qd 532 orders a c h cc k for radial or rotational decel- 

central finger velocities during a hand scaling adds up to a eration 534 dutring motions prior to finger liftoff. The 

net vertical translation. Similar phenomena occur during method for detecting radial or rotational deceleration is the 

hand rotations, except that if the rotation is centered at the same ^ lnat detailed in the description of translation extrac- 

wrist with forearm fixed rather than centered at the tion. 

forepalms, a net horizontal translation will appear in the sum 20 FIG. 37 shows the details of hand translational velocity 

of motions from any combination of fingers. measurements referred to in process 506 of FIG. 34. The 

Since the differences in finger motion are usually greatest simplest way to compute a hand translation velocity would 

between thumb and pinky, step 526 only retrieves the current be to simply avcragc me laleral vclocitics of cac h finger, 

and previous positions of the innermost and outermost However, the user expects the motion or control to display 

touching fingers for the hand scaling and rotation measure- 25 gain t0 be musUnt regardless of how many fingers are beill g 

mc ? s ' MO . • « « t . . moved, even if some are resting stationary. Furthermore, if 

Step 528 then computes Ac hand scahng ve ocity ^ ^ ^ fc ^ullineously scaling or routing the hand, a 
from the change in distance between the innermost ringer FI . & t . 

and outermost finger FO with approximately the following SUD P'* K avera « e » * ?P unous net 

equation* 30 cause<1 bv uncanceled central finder motions. 

Therefore, in a preferred embodiment the translational 

d{Fl[n} y FOjn]) - d(Ft[n - ll FO[n - 1]) (66) component extractor carefully assigns weightings for each 

w "*^ " & finger before computing the average translation. Step 540 

initializes the translation weighting Fi^ of each finger to its 

where d(FI[n] > FO[n]) is the squared Euclidean distance 35 conlact P™* ; ic - ^n]«FiIn] ^is ensures that 

between the fingers: fingers not touching the surface do not dilute the average 

with their zero velocities and that fingers which only touch 

d(Fi[ nLFOfn])- V(FIJ n}- FOjnD 2 +(Fiy[ n]- FO y ( nj) 2 (67) lightly have less influence since their position and velocity 

measurements may be less reliable. The next step 544 

If one of the innermost or outermosrfingers" was not 40 "decreases the weightings of "fingers which are relatively ' 

touching during the previous proximity image, the change in stationary so that the control to display gain of intentionally 

separation is assumed to be zero. Similarly, step 530 com- moving fingers is not diluted. This can be done by finding 

putes the hand rotational velocity H vr from the change in me fastest moving recording its speed as a maximum 

angle between the innermost and outermost finger with fiQger ^ each fi > s KanslXion weighting 

approximately the following equation: 45 ^ proportion to ^ speed by me maximurn of the 

finger speeds, as shown approximately in the formula below: 

W w [n]= (68) 



( L(FI[n], FO[n]) - L(Fl[n - 1], FO[n - 1 ] >| ( d(Ft[n] y FOW )} / Fw[n] V*" (69) 



The change in angle is multiplied by the current separa- 
tion to convert it to the same units as the translation and 
scaling components. These equations capture any rotation 
and scaling components of hand motion even if the hand is 55 
also translating as a whole, thus making the rotation and 
scaling degrees of freedom integral with translation. 

Another reason the computations above are restricted to 
the thumb and pinky or innermost and outermost fingers is 
that users may want to make fine translating manipulations 60 
with the central fingers, i.e. index, middle, and ring, while 
the thumb and pinky remain stationary. If changes in dis- 
tances or angles between the central fingers and the thumb 
were averaged with Equations 66-68, this would not be 
possible because central finger translations would cause the 65 
appearance of rotation or scaling with respect to the station- 
ary thumb or pinky. However, Equations 56-60 applied in 



where the power ptw adjusts the strength of the speed 
dependence. Note that step 544 can be skipped for applica- 
tions such as computer-aided-design in which users desire 
both a normal cursor motion gain mode and a low gain 
mode. Lower cursor motion gain is useful for fine, short 
range positioning, and would be accessed by moving only 
one or two ringers while keeping the rest stationary. 

Step 546 decreases the translation weightings for the 
central fingers during hand scalings and rotations, though it 
doesn't prevent the central fingers from making fine trans- 
lational manipulations while the thumb and pinky are sta- 
tionary. The formulas below accomplish this seamlessly by 
downscaling the central translation weightings as the mag- 
nitudes of the rotation and scaling velocities become sig- 
nificant compared to Yi polanHresh : 



60 
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that the decision after liftoff to continue cursor motion 

- Fi *^ x ******* depends on the state of the deceleration flag before liftoff 

Kpoh»/w*3h + iw*r[n]| began. The final step 560 updates the autoregressive or 

r , ^ moving window average of the hand translation velocity 

Fi^[n) * * 7^r^i» ^ 5 vcctor > which can become the velocity of continued cursor 

Kvo*«nH+\H„in\\ + \H„m motion after MofL Acmal gcneration Qf tfac continue 

cursor motion signals occurs in the chord motion recognizer 

where these equations are applied only to the central fingers 18 as will be discussed with FIG. 40. 

whose identities i are between the innermost and outermost. Note that this cursor motion continuation method has 

Note that since hand scaling does not cause much horizontal 10 several advantages over motion continuation methods in 

translation bias, the horizontal translation weighting Fi^Jn] related art. Since the decision to continue motion depends on 

need not be affected by hand scaling velocity HJp], as a percentage acceleration which inherently normalizes to 

indicated by the lack of a hand scaling term in Equation 70. any speed range, the user can intentionally invoke motion 

The translation weightings of the innermost and outermost continuation from a wide range of speeds including very low 

fingers are unchanged by the polar component speeds, i.e., 15 speeds. Thus the user can directly invoke slow motion 

FI^n^n^JnhFUn] and FO v »FOvJoKO w continuation to auto scroll a document at readable speeds, 

[n]. Step 548 finally computes the hand translation velocity This is not true of Watanabe's method in U.S. Pat. No. 

vector (H JnpIJnD from the weighted average of the 4,734,685, which only continues motion when the user's 

finger velocities: motion exceeds a high speed threshold, nor of Logan et al.'s 

20 method in U.S. Pat. No. 5,327,161, which if enabled for low 

5 (72) finger speeds will undesirably continue motion when a user 

2j Fi^r^ decelerates on approach to a large target but fails to stop 

ffrxl"] = - completely before lifting off. Percentage acceleration also 

£ FKwx captures user intent more clearly than position of a finger in 

' _1 25 a border area. Position of a finger in a border area as used 

5 m ^.S. Pat No. 5,543,591 to Gillespie et al. is ambiguous 

^ Flirty, because the cursor can reach its desired target on the display 

H m just as the finger enters the border, yet the touchpad device 

v A t will continue cursor motion past the target because it thinks 

2-j Fivwy 30 the finger has run out of space to move. In the present 

invention, on the other hand, the acceleration ratio will 
remain near one if the ringers can slide off the edge of the 

The last part of the translation calculations is to test for the sensing array without hitting a physical barrier, sensibly 

lateral deceleration of the fingers before liftoff, which reli- invoking motion continuation. But if the fingers decelerate 

ably indicates whether the user wishes cursor motion to stop 35 before crossing or stop on the edge of the sensing array, the 

at liftoff. If deceleration is not detected prior to liftoff, the cursor will stop as desired. 

user may intend cursor motion to continue after liftoff, or the The details of the differential hand pressure extraction 
user may intend a special "one-shot" command to be process 508 are shown in FIG. 38. Fingertip proximity 
invoked. Decision diamond 550 only invokes the decelera- quickly saturates when pressure is applied through the bony 
tion tests while finger -proximities are- not- dropping- too 40- tip- normal to a hard surface/ Unless the surface itself is • 
quickly to prevent the perturbations in finger centroids highly compliant, the best dynamic range of fingertip pres- 
which can accompany finger liftoff from interfering with the sure is obtained with the fingers outstretched and hand 
deceleration measurements. Step 551 computes the percent- nearly flattened so that the compressible soft pulp under- 
age acceleration or ratio of current translation speed [(H^ neath the fingertips rests on the surface. Decision diamond 
[nJjH^n])! to a past average translation speed preferably 45 562 therefore causes the tilt and roll hand pressure compo- 
computed by a moving window average or autoregressive nents to be set to zero in step 564 and pressure extraction to 
filter. Decision diamond 552 causes the translation dcccl- abort unless the hand is nearly flattened. Inherent in the test 
eration flag to be set 556 if the acceleration ratio is less than for hand flattening 562 is a finger count to ensure that most 
a threshold. If this threshold is set greater than one, the user of the five ringers and both palm heels are touching the 
will have to be accelerating the fingers just prior to liftoff for 50 surface to maximize the precision of the hand pressure 
cursor motion to continue. If the threshold is set just below measurements, though technically only three non-collinear 
one, cursor motion will reliably be continued as long as the hand contacts arranged like a tripod are necessary to estab- 
user maintains a constant lateral speed prior to liftoff, but if lish tilt and roll pressures. Decision diamond 562 can also 
the user begins to slow the cursor on approach to a target require the user to explicitly enable three-dimensional 
area of the display the deceleration flag will beset. Decision 55 manipulation with an intuitive gesture such as placing all 
diamond 554 can also cause the deceleration flag to be set if five fingers on the surface, briefly tapping the palm heels on 
the current translation direction is substantially different the surface, and finally resting the palm heels on the surface, 
from an average of past directions. Such change in direction Decision diamond 566 causes step 568 to capture and store 
indicates the hand motion trajectory is curving, in which reference proximities for each contact path when the prox- 
case cursor motion should not be continued after liftoff 60 imity of all contacts have stabilized at the end of this 
because accurately determining the direction to the user's initiation sequence. The tilt and roll pressure components are 
intended target becomes very difficult. If neither decelera- again zeroed 564 for the sensor array scan cycle during 
tion nor curved trajectories are detected, step 558 clears the which this calibration is performed, 
translation deceleration flag. This will enable cursor motion However, during subsequent scan cycles the user can tilt 
continuation should the fingers subsequently begin liftoff. 65 the hand forward applying more pressure to the fingertips or 
Note that decision diamond 550 prevents the state of the backward applying more pressure to the palm heels, or the 
translation deceleration flags from changing during liftoff so user can roll the hand outward onto the pinky and outer palm 
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heel or inward applying more pressure to the thumb, index -continued 

finger and inner palm heel. Step 570 will proceed to calcu- . H ^ ^ 

late an unweighted average of the current contact positions. MM: - HM A^ dom^amspeed ) 
Step 572 computes for each band part still touching the 

surface the ratio of current proximity to the reference 5 

proximity previously stored. To make these ratios less wncrc H^^Jn], H^n], and H rj ^J>] are autoregres- 

sensitive to accidental lifting of hand parts, step 574 clips sivc averages over time of the translation speed, scaling 

them to be greater or equal to one so only increases in s P ced ^ md rotational speed, where: 

proximity and pressure register in the tilt and roll measure- dominant^spccd-maxCH^^Jn], H^^n], H rwpe j[nJ} (78) 

ments. Another average contact path position is computed in 10 , , , , , , _ , A , . 

step 576, but this one is weighted by the above computed and wh ? re P<b oonuob the strength of the filter. As pete is 

proximity ratios for each path. The difference between these awards infinity the dominant component is picked 

weighted and unweighted contact position averages taken in out and a " components less than the dominant tend toward 

step 578 produces a vector whose direction can indicate the ^ro, producing the orthogonal cursor effect well-known in 

direction of roll or tilt and whose magnitude can control the 15 J™"* W^Uons^As pds is adjusted towards zero the 

rate of roll or tilt about x and y axes. filtcrs havc no cffe <?- ^ably, pds is set in between so that 

. , , . , components significantly slower than the dommant are 

Since the weighted and unweighted position averages are sk)wed but close to me dommant m 

only influenced by positions of currently contacting fingers ^ ^ barel affecte< ^ preserving me p^mty of 

and increases m contact pressure or proximity, the method is n di ^ modoD fa mul ,^, e d of £reedon) „ Qnce ^ 

insensitive to finger liftoffs. Computation of reference- autoregressive averaging helps to pick out the component or 

normalized proxnmty ratios in step 572 rather than absolute mmpon&ats which ue dominant over the long term and 

changes m proxjmity prevents the large palm heel contacts ^ otheis even while the dominant components are 

from having undue influence on the weighted average posi- slowing to a stop 

k° n ' 25 Step 512 takes a second pass with a related filter known 

Since only the current contact positions are used in the as a dead-zone filter. A dead-zone filter produces zero output 

average position computations, the roll and tilt vector is velocity for input velocities less than a speed threshold but 

independent of lateral motions such as hand translation or produces output speeds in proportion to the difference 

rotation as long as the lateral motions don't disturb finger between the input speed and the threshold for input veloci- 

pressure, thus once again achieving integrality. However, 30 ties that exceed the threshold. Preferably the speed threshold 

hand scaling and differential hand pressure are difficult to or width of the dead zone is set to a fraction of the maximum 

use at the same time because flexing the fingers generally c f current component speeds. All velocity components are 

causes significant decreases in fingertip contact area and filtered using this same dead zone width. The final extracted 

thus interferes with inference of fingertip pressure changes. component velocities are forwarded to the chord motion 

When this becomes a serious problem, a total hand pressure 35 recognizer module 18 which will determine what if any 

component can be used as a sixth degree of freedom in place input events should be generated from the motions, 

of the hand scaling component. This total pressure compo- fig. 39A shows the details of the finger synchronization 

nent causes cursor velocity along a z-axis in proportion to detector module 14. The synchronization detection process 

deviations of the average of the contact proximity ratios described below is repeated for each hand independently, 

from one. Alternative embodiments may include further 40 step 600 fetches proximity markers and identifications for - 

enhancements such as adapting the reference proximities to the hand's current paths. The identifications will be neces- 

slow variations in resting hand pressure and applying a dead sary to ignore palm paths and identify combinations of 

zone filter to ignore pressure difference vectors with small synchronized fingers, while the proximity markers record 

magnitudes. the time at which each contact path first exceeds a press 

Despite the care taken to measure the polar velocity, 45 proximity threshold and the time at which each contact path 

translation velocity, and hand pressure components in such drops below a release proximity threshold prior to total 

a way that the resultant vectors are independent of one liftoff. Setting these proximity thresholds somewhat higher 

another, uneven finger motion during hand scaling, rotation, than the minimum proximity considered significant by the 

or translation can still cause minor perturbations in mea- segmentation search process 264, produces more precise 

sure ments of one degree of freedom while primarily 50 finger press and release times. 

attempting to move in another. Non-linear filtering applied Step 603 searches for subsets of fingers which touch down 

in steps 510 and 512 of FIG. 34 removes the remaining at about the same time and for subsets of fingers which lift 

motion leakage between dominant components and nearly off at about the same time. This can be done by recording 

stationary components. In steps 510 each component veloc- each finger path along with its press time in a temporally 

ity is downscaled by the ratio of its average speed to the 55 ordered list as it crosses the press proximity threshold Since 

maximum of all the component speeds, the dominant com- the primary function of the palms is to support the forearms 

ponent speed: while the hands are resting, palm activity is ignored by the 

typing 12 and chord motion recognizers 18 except during 

(H xyapttd [n\ y* (74) differential hand pressure extraction, and palm heel presses 

dominanispeed j 60 can be excluded from this list and most other synchroniza- 
tion tests. To check for synchronization between the two 

H [ rt j : = H j rt j f Mrj«p»«f[rt] Y*** C 75 ) most recent finger presses, the press times of the two most 

* v {dominamspeed ) recent entries in the list are compared. If the difference 

H . ^ between their press times is less than a temporal threshold, 

MM'. = MM x { ^ m ^^ eed ) 65 ^ e two ^° er P resses are considered synchronized. If not, the 

pee most recent finger press is considered asynchronous. Syn- 
chronization among three or more fingers up to five is found 
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by comparing press times of the three, four, or five most will eventually release the queue hold by having step 615 

recent list entries. If the press time of the most recent entry delete the. synchronized presses from the keypress queue if 

is within a temporal threshold of the nth most recent entry, both fingers remain touching a long time. Though this aborts 

synchronization among the n most recent finger presses is the hypothesis that the presses were intended as key taps, the 

indicated. To accommodate imprecision in touchdown 5 presses are also less likely to be key taps if the fingers aren't 

across the hand, the magnitude of the temporal threshold lifted soon after touchdown. 

should increase slightly in proportion to the number of If the synchronized fingers are not lifting, decision dia- 
fingers being tested for synchronization. The largest set of mond 616 leaves the synchronization marker pending so 
recent finger presses found to be synchronized is recorded as synchronization checks can be continued with updated path 
the synchronized subset, and the combination of finger 10 parameters 600 after the next scan cycle. If the synchronized 
identities comprising this subset is stored conveniently as a fingers are lifting, but decision diamond 618 finds with the 
finger identity bitfield. The term subset is used because the help of the synchronization release search 603 that they are 
synchronized press subset may not include all fingers cur- doing so asynchronously 618, step 622 releases any holds on 
rently touching the surface, as happens when a finger the keypress queue assuming any synchronized finger pair 
touches down much earlier man me other fingers yet remains 15 was intended to be two keypresses. Though the synchro- 
touching as they simultaneously touch down. An ordered list nized finger presses are not deleted from the keypress queue 
of finger release times is similarly maintained and searched at this point, they may have already been deleted in step 612 
separately. Alternative embodiments may require that a if me pressed subset contained more than two. Also, step 624 
finger still be touching the surface to be included in the clears the temporal synchronization marker, indicating that 
synchronized press subset 20 no further synchronization tests need be done for this subset. 

Decision diamond 602 checks whether a synchronization Continuing to FIG. 39B, if the fingers synchronized 
marker is pending from a previous image scan cycle. If not, during touchdown also lift simultaneously, step 618 removes 
decision diamond 604 checks whether the search 603 found them and any holds from the keypress queue in case they 
a newly synchronized press subset in the current proximity were a pair awaiting a positive release synchronization test, 
image. If so, step 606 sets the temporal synchronization 25 Further tests ensue to detemine whether the synchronized 
marker to the oldest press within the new synchronized fingers meet additional chord tap conditions. As with single 
subset. Additional finger presses may be added to the subset finger taps, the synchronized fingers cannot be held on the 
during future scan cycles without affecting the value of this surface more than about half a second if they are to qualify 
temporal synchronization marker. If there is currently no as a chord tap. Decision diamond 626 tests this by thresh- 
finger press synchronization, decision diamond 605 deter- 30 olding the time between the release of the last remaining 
mines whether three or more fingers have just been released synchronized finger and the temporal press synchronization 
simultaneously. Simultaneous release of three or more fin- marker. A chord tap should also exhibit a limited amount of 
gers should not occur while typing with a set of fingers but lateral finger motion, measured either as an average of peak 
does occur when lifting fingers off the surface from rest. finger speeds or distance traveled since touchdown in deci- 
Therefore simultaneous release of three or more fingers 35 sion diamond 628. If the quick release and limited lateral 
reliably indicates that the released fingers are not intended as motion conditions are not met, step 624 clears the synchro- 
keypresses and should be deleted from the keypress queue nization marker with the conclusion that the synchronized 
606, regardless of whether these same fingers touched down fingers were either just resting fingers or part of a chord 
synchronously. Release synchronization of two fingers is not slide. 

by itself a reliable indicator of typing intent and has no effect 40 If the chord tap conditions are met, step 630 looks up, - 

on the keypress queue. The keypress queue is described later using the synchronized subset bitfield, any input events such 

with FIGS. 42-43B. as mouse clicks or keyboard commands assigned to the 

Once a press synchronization marker for the hand is combination of fingers in the chord tap. Some chords such 

pending, further processing checks the number of finger as those including all four fingertips may be reserved as 

presses which are synchronized and waits for release of the 45 resting chords 634, in which case decision diamond 632 will 

synchronized fingers. If decision diamond 608 finds three or find they have no associated input events. If the chord does 

more fingers in the synchronized press subset, the user can't have tap input events, step 636 appends these to the main 

possibly be typing with these fingers. Therefore step 612 outgoing event queue of the host communication interface 

immediately deletes the three or more synchronized presses 20. Finally step 624 clears the synchronization marker in 

from the keypress queue. This way they cannot cause key 50 readiness for future finger synchronizations on the given 

symbol transmission to the host, and transmission of key hand. 

symbols from subsequent asynchronous presses is not As a further precaution against accidental generation of 

blocked waiting for the synchronized fingers to be released. chord taps while typing, it is also useful for decision 

However, when the synchronization only involves two diamond 632 to ignore through step 634 the first chord tap 

finger presses 608, it is difficult to know whether the user 55 which comes soon after a valid keypress without a chord 

intended to tap a finger pair chord or intended to type two slide in between. Usually after typing the user will need to 

adjacent keys and accidentally let the key presses occur reposition the mouse cursor before clicking, requiring an 

simultaneously. Since such accidental simultaneous presses intervening chord slide. If the mouse cursor happens to 

are usually followed by asynchronous releases of the two already be in place after typing, the user may have to tap the 

fingers, but finger pair chords are usually released 60 finger chord a second time for the click to be sent, but this 

synchronously, the decision whether the presses are asyn- is less risky than having an accidental chord tap cause an 

chronous key taps or chord taps must be delayed until finger unintended mouse button click in the middle of a typing 

release can be checked for synchronization. In the session. 

meantime, step 610 places a hold on the keypress queue to FIG. 40A shows the detailed steps of the chord motion 
prevent transmission of key symbols from the possible 65 recognizer module 18. The chord motion recognition pro- 
finger chord or any subsequent finger presses. To prevent cess described below is repeated for each hand indepen- 
long backups in key transmission, decision diamond 614 dently. Step 650 retrieves the parameters of the hand's 
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identified paths 250 and the hand's extracted motion com- 
ponents from the motion extraction module 16. If a slide of 
a finger chord has not already started, decision diamond 652 
orders slide initiation tests 654 and 656. To distinguish slides 
from glancing finger taps during typing, decision diamond 5 
654 requires at least two fingers from a hand to be touching 
the surface for slide mode to start. There may be some 
exceptions to this rule such as allowing a single finger to 
resume a previous slide within a second or so after the 
previous slide chord lifts off the surface. 10 

In a preferred embodiment, the user can start a slide and 
specify its chord in either of two ways. In the first way, the 
user starts with the hand floating above the surface, places 
some fingers on the surface possibly asynchronously, and 
begins moving all of these fingers laterally. Decision dia- IS 
mond 656 initiates the slide mode only when significant 
motion is detected in all the touching fingers. Step 658 
selects the chord from the combination of fingers touching 
when significant motion is detected, regardless of touch- 
down synchronization. In this case coherent initiation of 20 
motion in all the touching fingers is sufficient to distinguish 
the slide from resting fingers, so synchronization of touch- 
down is not necessary. Also, novice users may erroneously 
try to start a slide by placing and sliding only one finger on 
the surface, forgetting that multiple fingers are necessary. 25 
Tolerance of asynchronous touchdown allows them to seam- 
lessly correct this by subsequently placing and sliding the 
rest of the fingers desired for the chord. The slide chord will 
then initiate without forcing the user to pick up all fingers 
and start over with synchronized finger touchdowns. 30 

In the second way, the user starts with multiple fingers 
resting on the surface, lifts a subset of these fingers, touches 
a subset back down on the surface synchronously to select 
the chord, and begins moving the subset laterally to initiate 
the slide. Decision diamond 656 actually initiates the slide 35 
mode when it detects significant motion in all the fingers of 
the synchronized subset. Whether the fingers which 
remained resting on the surface during this sequence begin 
to move does not matter since in this case the selected chord 
is determined in -step 658-by* the-combination of fingers in 40 
the synchronized press subset, not from the set of all 
touching fingers. This second way has the advantage that the 
user does not have to lift the whole hand from the surface 
before starting the slide, but can instead leave most of the 
weight of the hands resting on the surface and only lift and 45 
press the two or three fingers necessary to identify the most 
common finger chords. 

To provide greater tolerance for accidental shifts in rest- 
ing finger positions, decision diamond 656 requires both that 
all relevant fingers are moving at signilcant speed and that 50 
they are moving about the same speed. This is checked either 
by thresholding the geometric mean of the finger speeds or 
by thresholding the fastest finger's speed and verifying that 
the slowest finger's speed is at least a minimum fraction of 
the fastest finger's speed. Once a chord slide is initiated, step 55 
660 disables recognition of key or chord taps by the hand at 
least until either the touching fingers or the synced subset 
lifts off. 

Once the slide initiates, the chord motion recoginizer 
could simply begin sending raw component velocities paired 60 
with the selected combination of finger identities to the host. 
However, in the interest of backward compatibility with the 
mouse and key event formats of conventional input devices, 
the motion event generation steps in FIG. 40B convert 
motion in any of the extracted degrees of freedom into 65 
standard mouse and key command events which depend on 
the identity of the selected chord. To support such motion 
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conversion, step 658 finds a chord activity structure in a 
lookup table using a bitfield of the identities of either the 
touching fingers or the fingers in the synchronized subset. 
Different finger identity combinations can refer to the same 
chord activity structure. In the preferred embodiment, all 
finger combinations with the same number of non-thumb 
fingertips refer to the same chord activity structure, so slide 
chord activities are distinguished by whether the thumb is 
touching and how many non-thumb fingers are touching. 
Basing chord action on the number of fingertips rather than 
their combination still provides up to seven chords per hand 
yet makes chords easier for the user to memorize and 
perform. The user has the freedom to choose and vary which 
fingertips are used in chords requiring only one, two or three 
fingertips. Given this freedom, users naturally tend to pick 
combinations in which all touching fingertips are adjacent 
rather than combinations in which a finger such as the ring 
finger is lifted but the surrounding fingers such as the middle 
and pinky must touch. One chord typing study found that 
users can tap these finger chords in which all pressed 
fingertips are adjacent twice as fast as other chords. 

The events in each chord activity structure are organized 
into slices. Each slice contains events to be generated in 
response to motion in a particular range of speeds and 
directions within the extracted degrees of freedom. For 
example, a mouse cursor slice could be allocated any 
translational speed and direction. However, text cursor 
manipulation requires tour slices, one for each arrow key, 
and each arrow's slice integrates motion in a narrow direc- 
tion range of translation. Each slice can also include motion 
sensitivity and so-called cursor acceleration parameters for 
each degree of freedom. These will be used to discretize 
motion into the units such as arrow key clicks or mouse 
clicks expected by existing host computer systems. 

Step 675 of chord motion conversion simply picks the 
first slice in the given chord activity structure for processing. 
Step 676 scales the current values of the extracted velocity 
components by the slice's motion sensitivity and accelera- 
tion parameters. Step 677 geometrically projects or clips the 
scaled velocity components into the- slice's defined-speed - 
and direction range. For the example mouse cursor slice, this 
might only involve clipping the rotation and scaling com- 
ponents to zero. But for an arrow key slice, the translation 
velocity vector is projected onto the unit vector pointing in 
the same direction as the arrow. Step 678 integrates each 
scaled and projected component velocity over time in the 
slice's accumulators until decision diamond 680 determines 
at least one unit of motion has been accumulated. Step 682 
looks up the slice's preferred mouse, key, or three- 
dimensional input event format, attaches the number of 
accumulated motion units to the event, and step 684 dis- 
patches the event to the outgoing queue of the host com- 
munication interface 20. Step 686 subtracts the sent motion 
events from the accumulators, and step 688 optionally clears 
the accumulators of other slices. If the slice is intended to 
generate a single key command per hand motion, decision 
diamond 689 will determine that it is a one-shot slice so that 
step 690 can disable further event generation from it until a 
slice with a different direction intervenes. If the given slice 
is the last slice, decision diamond 692 returns to step 650 to 
await the next scan of the sensor array. Otherwise step 694 
continues to integrate and convert the current motion for 
other slices. 

Returning to FIG. 40A, for some applications it may be 
desirable to change the selected chord whenever an addi- 
tional finger touches down or one of the fingers in the chord 
lifts off. However, in the preferred embodiment, the selected 
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chord cannot be changed after slide initiation by asynchro- 
nous finger touch activity. This gives the user freedom to rest 
or lift additional fingers as may be necessary to get the best 
precision in a desired degree of freedom. For example, even 
though the finger pair chord does not include the thumb, the 
thumb can be set down shortly after slide initiation to access 
the full dynamic range of the rotation and scaling degrees of 
freedom. In fact, all remaining lifted fingers can always be 
set down after initiation of any chord to allow manipulation 
by the whole hand. Likewise, all fingers but one can be 
lifted, yet translation will continue. 

Though asynchronous finger touch activity is ignored, 
synchronized lifting and pressing of multiple fingers subse- 
quent to slide initiation can create a new synchronized 
subset and change the selected chord. Preferably this is only 
allowed while the hand has paused but its fingers are still 
resting on the surface. Decision diamond 670 will detect the 
new subset and commence motion testing in decision dia- 
mond 673 which is analogous to decision diamond 656. If 
significant motion is found in all fingers of the newly 
synchronized subset, step 674 will select the new subset as 
the slide chord and lookup a new chord activity structure in 
analogy to step 658. Thus finger synchronization again 
allows the user to switch to a different activity without 
forcing the user to lift the whole hand from the surface. 
Integration of velocity components resumes but the events 
generated from the new chord activity structure will pre- 
sumably be different. 

It is advantageous to provide visual or auditory feedback 
to the user about which chord activity structure has been 
selected. This can be accomplished visually by placing a row 
of five light emitting diodes across the top of the multi-touch 
surface, with one row per hand to be used on the surface. 
When entering slide mode, step 658 would turn on a 
combination of these lights corresponding to the combina- 
tion of fingers in the selected chord. Step 674 would change 
the combination of active lights to match the new chord 
activity structure should the user select a new chord, and 
step 668 would turn them off. Similar lights could be 
emulated on the host computer display -24'rThe- lights could - 
also be flashed to indicate the finger combination detected 
during chord taps in step 636. The implementation for 
auditory feedback would be similar, except light combina- 
tions would be replaced with tone or tone burst combina- 
tions. 

The accumulation and event generation process repeats 
for all array scan cycles until decision diamond 664 detects 
liftoff by all the fingers from the initiating combination. 
Decision diamond 666 then checks the pre-liftoff decelera- 
tion flag of the dominant motion component. The state of 
this flag is determined by step 556 or 558 of translation 
extraction (FIG. 37) if translation is dominant, or by corre- 
sponding flags in step 534 of polar extraction. If there has 
been significant deceleration, step 668 simply exits the 
chord slide mode, setting the selected chord to null. If the 
flag indicates no significant finger deceleration prior to 
liftoff, decision diamond 666 enables motion continuation 
mode for the selected chord. While in this mode, step 667 
applies the pre-liftoff weighted average (560) of dominant 
component velocity to the motion accumulators (678) in 
place of the current velocities, which are presumably zero 
since no fingers touch the surface. Motion continuation 
mode does not stop until any of the remaining fingers not in 
the synchronized subset are lifted or more fingers newly 
touch down. This causes decision diamond 664 to become 
false and normal slide activity with the currently selected 
chord to resume. Though the cursor or scrolling velocity 
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does not decay during motion continuation mode, the host 
computer can send a signal instructing motion continuation 
mode to be canceled if the cursor reaches the edge of the 
screen or end of a document. Similarly, if any fingers remain 

5 on the surface during motion continuation, their translations 
can adjust the cursor or scrolling velocity. 

In the preferred embodiment, the chord motion recogniz- 
ers for each hand function independently and the input 
events for each chord can be configured independently. Tliis 

10 allows the system to allocate tasks between hands in many 
different ways and to support a variety of bimanual manipu- 
lations. For example, mouse cursor motion can be allocated 
to the fingertip pair chord on both hands and mouse button 
drag to a triple fingertip chord on both hands. This way the 

15 mouse pointer can be moved and drug with either hand on 
either half of the surface. Primary mouse clicks would be 
generated by a tap of a fingertip pair on either half of the 
surface, and double-clicks could be ergonomically generated 
by a single tap of three fingertips on the surface. Window 

20 scrolling could be allocated to slides of four fingers on either 
hand. 

Alternatively, mouse cursor manipulations could be allo- 
cated as discussed above to the right hand and right half of 
the surface, while corresponding text cursor manipulations 

25 are allocated to chords on the left hand. For instance, left 
fingertip pair movement would generate arrow key com- 
mands corresponding to the direction of motion, and three 
fingertips would generate shift arrow combinations for 
selection of text. 

30 For host computer systems supporting manipulations in 
three or more degrees of freedom, a left hand chord could be 
selected to pan, zoom, and rotate the display background 
while a corresponding chord in the right hand could 
translate, resize and rotate a foreground object. These chords 

35 would not have to include the thumb since the thumb can 
touch down anytime after initiating chord motion without i 
changing the selected chord. The user then need add the 
thumb to the surface when attempting rotation or scaling. 
Finger chords which initially include the thumb can be 

40 reserved for one-shot command gestures, which only gen 
erate input events once for each slide of a chord rather than 
repeating transmission each time an additional unit of 
motion is detected. For example, the common editing com- 
mands cut, copy and paste can be intuitively allocated to a 

45 pinch hand scaling, chord tap, and anti-pinch hand scaling of 
the thumb and an opposing fingertip. 

FIG. 41 shows the steps within the key layout definition 
and morphing process, which is part of tie typing recogni- 
tion module 12. Step 700 retrieves at system startup a key 

50 layout which has been pre -specified by the user or manu- 
facturer. The key layout consists of a set of key region data 
structures. Each region has associated with it the symbol or 
commands which should be sent to the host computer when 
the region is pressed and coordinates representing the loca- 

55 tion of the center of the region on the surface. In the 
preferred embodiment, arrangement of those key regions 
containing alphanumeric and punctuation symbols roughly 
corresponds to either the QWERTY or the Dvorak key 
layouts common on mechanical keyboards. 

so In some embodiments of the multi-touch surface appara- 
tus it is advantageous to be able to snap or morph the key 
layout to the resting positions of the bands. This is especially 
helpful for multi-touch surfaces which are several times 
larger than the standard keyboard or key layout, such as one 

65 covering an entire desk. Fixing the key layout in one small 
fixed area of such a surface would be inconvenient and 
discourage use of the whole available surface area. To 
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provide feedback to the user about changes in the position of right hand, and from the "A" key to the "F" key for the left 

the key layout, the position of the key symbols in these hand. Similarly, the thumbs can each be provided with a 

embodiments of the multi-touch surface would not be single oval-shaped depression at their default locations, 

printed permanently on the surface. Instead, the position of slanted slightly from vertical to match the default thumb 

the key symbols would be reprogrammably displayed on the 5 orientation. These would preferably correspond to "Space" 

surface by light emitting polymers, liquid crystal, or other and "Backspace" key regions for the right and left thumbs, 

dynamic visual display means embedded in the multi-touch respectively. Such minimal depressions can tactilely guide 

surface apparatus along with the proximity sensor arrays. users' hands back to home row of the key layout without 

Given such an apparatus, step 702 retrieves the current requiring users to look down at the surface and without 

paths from both hands and awaits what will be known as a 10 seriously disrupting finger chord slides and manipulations 

layout homing gesture. If decision diamond 704 decides on the surface. 

with the help of a hand's synchronization detector that all The positions of key regions off home row can be marked 

five of the hand's fingers have just been placed on the by other types of tactile indicators. Simply roughening the 

surface synchronously, step 706 will attempt to snap the key surface at key regions does not work well. Though humans 

layout to the hand such that the hand's home row keys lie IS easily differentiate textures when sliding fingers over them, 

under the synchronized fingertips, wherever the hand is on most textures cannot be noticed during quick taps on a 

the surface. Step 706 retrieves the measured hand offsets textured region. Only relatively abrupt edges or protrusions 

from the hand position estimator and translates all key can be sensed by the users' fingertips under typing condi- 

regions which are normally typed by the given hand in tions. Therefore, a small raised dot like a Braille dot is 

proportion to the measured hand offsets. Note the currently 20 formed on top of the surface at the center of each key region, 

measured rather than filtered estimates of offsets can be used The user receives feedback on the accuracy of their typing 

because when all five fingers are down there is no danger of strokes from where on the fingertip a dot is felt. This 

finger misidentification corrupting the measured offsets. feedback can be used to correct finger aim during future 

This procedure assumes thai the untranslated locations of the keypresses. Since single finger slides are ignored by the 

home row keys are the same as the default fnger locations 25 chord motion recognizer, the user can also slide a finger 

for the hand. around the surface in tactile search of a particular key 

Decision diamond 708 checks whether the fingers appear region's dot and then tap the key region when the dot is 

to be in a neutral, partially closed posture, rather closed than found, all without looking at the surface. Each dot should be 

outstretched or pinched together. If the posture is close to just large enough to be felt during tapping but not so large 

neutral, step 710 may further offset the keys normally typed 30 as to impede chord slides across the surface. Even if the dots 

by each finger, which for the most part are the keys in the are not large enough to impede sliding, they can still corrupt 

same column of the finger by the measured finger offsets. proximity and fingertip centroid measurements by raising 

Temporal filtering of these ringer offsets over several layout the fingertip flesh near the dot off the surface thus locally 

homing gestures will tend to scale the spacing between separating the flesh from the underlying proximity sensing 

columns of keys to the user's hand size. Spacing between 35 electrode. Therefore, in the preferred embodiment, the por- 

rows is scaled down in proportion to the scaling between tion of each dot above the surface dielectric is made of a 

columns. conductive material. This improves capacitive coupling 

With the key layout for the hand's keys morphed to fit the between the raised fingertip flesh and the underlying elec- 

size and current position of the resting hand, step 712 trades. 

updates the displayed position of the symbols on the surface, -40 — FIG. -42 shows the steps within the keypress detection- 
so that the user will see that the key layout has snapped to loop. Step 750 retrieves from the current identified path data 
the position of his hand. From this stage the user can begin 250 any paths which were recently created due to hand part 
to type and the typing recognizer 718 will use the morphed touchdown or the surface. Decision diamond 752 checks 
key region locations to decide what key regions are being whether the path proximity reached a keypress proximity 
pressed. The layout will remain morphed this way until 45 thresh for the first time during the current sensor array scan, 
either the user performs another homing gesture to move it If the proximity has not reached the threshold yet or has 
somewhere else on the surface, or until the user takes both already exceeded it previously, control returns to step 750 to 
hands off the surface for a while. Decision diamond 714 will try keypress detection on the next recent path. If the path just 
eventually time out so that step 716 can reset the layout to crossed the keypress proximity threshold, decision diamond 
its default position in readiness for another user or usage 50 754 checks whether the contact path has been identified as 
session. a finger rather than a palm. To give the users the freedom rest 
For smaller multi-touch surfaces in which the key layout the palms anywhere on the surface, palm presses should not 
is permanently printed on the surface, it is advantageous to normally cause keypresses, and are therefore ignored, 
give the user tactile feedback about the positions of key Assuming the path is a ringer, decision diamond 756 checks 
regions. However, any tactile indicators placed on the sur- 55 whether the hand the identified finger comes from is cur- 
face must be carefully designed so as not to impede smooth rently performing a chord slide gesture or writing via the pen 
sliding across the surface. For example, shallow depressions grip hand configuration. Asynchronous finger presses are 
made in the surface near the center of each key mimicking ignored once these activities have started, as also indicated 
the shallow depressions common on mechanical keyboard in step 660 of FIG. 40 A. Assuming such hand activities are 
keycaps would cause a vibratory washboard effect as the 60 not ongoing, decision diamond 757 proceeds with debounce 
hand slides across the surface. To minimize such washboard tests which check that the finger has touched the surface for 
effects, in the preferred embodiment the multi-touch surface at least two sensor array scan cycles and that it had been off 
provides for the fingertips of each hand a single, continuous the surface for several scan cycles before touching down, 
depression running from the default index fingertip location The path tracking module (FIG. 22) facilitates such liftoff 
to the default pinky fingertip location. This corresponds on 65 debouncing by reactivating in step 334 a finger's old path if 
the QWERTY key layout to shallow, slighdy arched chan- the finger lifts off and quickly touches back down over the 
nels along home row from the "J" key to the ";" key for the same spot. Upon reactivation the time stamp of the last 
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liftoff by the old path must be preserved for comparison with When a keypress is transmitted to the host it is advanta- 
ge time stamp of the new touchdown. geous for a sound generation device on the multi-touch 
If all of these tests arc passed, step 758 looks up the surface apparatus or host computer to emit an audible click 
current path position (Pjn], P >I Q ])» and step 760 finds me or beep as feedback to the user. Generation of audible click 
key region whose reference position is closest to the finger- 5 a nd beep feedback in response to keypresses is well known 
tip centroid. Decision diamond 762 checks that the nearest in co mmercial touchscreens, kiosks, appliance control 
region is i within a reasonable distance of the finger ,and if not cls> ^ mcchanica i kcyb oards in which the keyswitch 
causes the finger press to be i^ored Assuming a key region actioQ ^ neafl ^ ^ does nQt faave a make force 

L'cf , . J- L *H re h ^ rcSS h fi meDt *«hoM which feels distinctive to the user Feedback can 

data structure containing the path mdex identifier and finger „ , . ... .... it _ * . 

identity, the closest key Region, and a time stamp indicatmg 10 als ° be pro * lc ! ed fl ** a h ^ on nuilh-touch surface 
when the finger crossed the keypress proximity threshold. apparatus w *"ch flashes each time a keypress is sent. Key- 
Step 766 then appends this element data structure to the tail P"* 5 ** accompanied by modifier keypresses should cause 
of a FIFO keypress queue. This accomplished, processing lon & T flasnes or tones to acknowledge that the key symbol 
returns to step 750 to process or wait for touchdowns by includes modifiers. 

other fingers. 15 If the finger has not yet lifted, decision diamond 786 
The keypress queue effectively orders finger touchdowns checks whether its associated key region is a modifier such 
by when they pass the keypress proximity threshold. It thus as <shift>, <ctrl>, or <alt>. If so, step 788 advances to the 
fixes the order in which key symbols from each finger tap next element in the queue without deleting the head. Pro- 
will be transmitted to the host. However, an element's key cessing will continue at step 772 to see if the next element 
symbol is not assured transmission of the host once in the 20 is a valid key tap. If the next element successfully reaches 
keypress queue. Any of a number of conditions such as the transmission stage, step 784 will scan back toward the 
being part of a synchronized subset of pressing fingers can head of the queue for any modifier regions which are still 
cause it to be deleted from the queue before being trans- pressed. Then step 784 can send the next element's key 
mitted to the host. In this sense the keypress queue should symbol along with the modifying symbols of any preceding 
be considered a keypress candidate queue. Unlike the 25 modifier regions. 

ordered lists of finger touchdowns and releases maintained Decision diamond 782 requires that users touch the finger 
for each hand separately in the synchronization detector, the on the surface and lift back off within a few hundred 
keypress queue includes and orders the finger touchdowns milliseconds for a key to be sent. This liftoff timing require- 
from both bands. ment substitutes for the force activation threshold of 
FIG. 43A shows the steps within the keypress acceptance 30 mechanical keyswitches. Like the force threshold of 
and transmission loop. Step 770 picks the element at the mechanical keyswitches, the timing constraint provides a 
head of the keypress queue, which represents the oldest way for the user to rest the finger on the key surface without 
finger touchdown which has neither been deleted from the invoking a keypress. TThe synchronization detector 14 pro- 
queue as an invalid keypress candidate nor transmitted its vides another way for fingers to rest on the surface without 
associated key symbol. Decision diamond 772 checks 35 generating key symbols: they must touch down at the same - 
whether the path is still identified as a finger. While waiting time as at least one other finger. However, sometimes users . 
in the queue path proximity could have increased so much will start resting by simultaneously placing the central 
that the identification system decides the path is actually fingertips on the surface, but then they follow asynchro- 
from a palm heel, in which case step 778 deletes the nously with the pinky a second later and the thumb a second 
- keypress element without transmitting- to the host and step 40 -after that- These latter presses are'essentially asynchronous 
770 advances processing to the next element. Decision and won't be invalidated by the synchronization detector, 
diamond 774 also invalidates the element if its press hap- but as long as they are not lifted within a couple hundred 
pened synchronously with other fingers of the same hand. milliseconds, decision diamond 782 will delete them with- 
Thus decision diamond 774 follows through on deletion out transmission. But while decision diamond 782 provides 
command steps 601, 612, 615, 620 of the synchronization 45 tolerance of asynchronous finger resting, its requirement that 
detection process (FIG. 39). Decision diamond 776 invali- fingers quickly lift off, Le. crisply tap, the surface to cause 
dates the keypress if too much lateral finger motion has key generation makes it very difficult to keep most of the 
occurred since touchdown, even if that lateral finger motion fingers resting on the surface to support the hands while 
has not yet caused a chord slide to start. Because users may tapping long sequences of symbols. This causes users to 
be touch typing on the surface, several millimeters of lateral 50 raise their hands off the surface and float them above the 
motion are allowed to accommodate glancing fingertip surface during fast typing sequences. This is acceptable 
motions which often occur when quickly reaching for keys. typing posture except that the users arms will eventually tire 
This is much more glancing tap motion than is tolerated by if the user fails to rest the hands back on the surface between 
touebpads which employ a single finger slide for mouse sequences. 

cursor manipulation and a single finger tap for key or mouse 55 To provide an alternative typing posture which does not 

button click emulation. encourage suspension of the hands above the surface, deci- 

Decision diamond 780 checks whether the finger whose . sion diamond 790 enables a second key acceptance mode 

touchdown created the keypress element has since lifted off which does not require quick finger liftoff after each press, 

the surface. If so, decision diamond 782 checks whether it Instead, the user must stall with all five fingers of a hand 

was lifted off soon enough to qualify as a normal key tap. If 60 resting on the surface. Then each time a finger is asynchro- 

so, step 784 transmits the associated key symbol to the host nously raised off the surface and pressed on a key region, 

and step 778 deletes it from the head of the queue. Note that that key region will be transmitted regardless of subsequent 

a keypress is always deleted from the queue upon liftoff, but liftoff tuning. If the surface is hard such that fingertip 

even though it may have stayed on the surface for a time proximity quickly saturates as force is applied, decision 

exceeding the tap timeout, it may have still caused trans- 65 diamond 792 checks the impulsivity of the proximity profile 

mission as a modifier key, as an impulsive press with hand for how quickly the finger proximity peaks. If the proximity 

resting, or as a typematic press, as described below. profile increases to its peak very slowly over time no key 
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will be generated. This allows the user to gently set down a 
raised finger without generating, a key in case the user lifts 
the finger with the intention of generating a key but then 
changes his mind. If the touch surface is compressible, 
decision diamond 792 can more directly infer finger force 5 
from the ratio of measured fingertip proximity to ellipse axis 
lengths. Then it can threshold the inferred force to distin- 
guish deliberate key presses from gentle finger rests. Since 
when intending to generate a key the user will normally 
press down on the new key region quickly after lifting off the 10 
old key region, the impulsivity and force thresholds should 
increase with the time since the finger lifted off the surface. 

Emulating typematic on a multi-touch surface presents 
special problems if finger resting force cannot be distin- 
guished reliably from sustained holding force on a key 15 
region. In this case, the special touch riming sequence 
detected by the steps of FIG. 43B supports reliable type- 
matic emulation. Assuming decision diamond 798 finds that 
typematic hasn't started yet, decision diamond 794 checks 
whether the keypress queue element being processed repre- 20 
sents the most recent ringer touchdown on the surface. If any 
finger touchdowns have followed the touchdown repre- 
sented by this element, typematic can never start from this 
queue element. Instead, decision diamond 796 checks 
whether the element's finger has been touching longer than 25 
the normal tap time out. If the finger has been touching too 
long, step 778 should delete its keypress element because 
decision diamond 786 has determined it is not a modifier and 
decision diamond 794 has determined it can never start 
typematic. If decision diamond 794 determines that the 30 
keypress element does not represent the most recent 
touchdown, yet decision diamond 796 indicates the element 
has not exceeded the tap time out, processing returns to step 
770 to await either liftoff or timeout in a future sensor array 
scan. This allows finger taps to overlap in the sense that a 35 
new key region can be pressed by a finger before another 
finger lifts off the previous key region. However, either the 
press times or release times of such a pair of overlapping 
finger taps must be asynchronous to prevent the pair from 
-being considered a chord- tap. — - -40- 

Assuming the finger touchdown is the most recent, deci- 
sion diamond 800 checks whether the finger has been 
touching for a typematic hold setup interval of between 
about half a second and a second. If not, processing returns 
to 770 to await either finger liftoff or the hold setup 45 
condition to be met during future scans of the sensor array. 
When the hold setup condition is met, decision diamond 802 
checks whether all other fingers on the hand of the given 
finger keypress lifted off the surface more than a half second 
ago. If they did, step 804 will initialize typematic for the 50 
given keypress element. The combination of decision dia- 
monds 800 and 802 allow the user to have other fingers of 
the hand to be resting on the surface when a finger intended 
for typematic touches down. But typematic will not start 
unless the other fingers lift off the surface within half a 55 
second of the desired typematic finger's touchdown, and 
typematic will also not start until the typematic finger has a 
continued to touch the surface for at least half a second after 
the others lifted off the surface. If these stringent conditions 
are not met, the keypress element will not start typematic 60 
and will eventually be deleted through either tap timeout 782 
when the finger lifts off or through tap timeout 796) if 
another touches down after it. 

Step 804 simply sets a flag which will indicate to decision 
diamond 798 during future scan cycles that typematic has 65 
already started for the element. Upon typematic 
initialization, step 810 sends out the key symbol for the first 
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time to the host interface communication queue, along with 
any modifier symbols being held down by the opposite hand. 
Step 812 records the time the key symbol is sent for future 
reference by decision diamond 808. Processing then returns 
to step 770 to await the next proximity image scan. 

Until the finger lifts off or another taps asynchronously, 
processing will pass through decision diamond 798 to check 
whether the key symbol should be sent again. Step 806 
computes the symbol repeat interval dynamically to be 
inversely proportional to finger proximity. Thus the key will 
repeat faster as the finger is pressed on the surface harder or 
a larger part of the fingertip touches the surface. This also 
reduces the chance that the user will cause more repeats than 
intended since as finger proximity begins to drop during 
liftoff the repeat interval becomes much longer. Decision 
diamond 808 checks whether the dynamic repeat interval 
since the last typematic symbol send has elapsed, and if 
necessary sends the symbol again in 810 and updates the 
typematic send time stamp 812. 

It is desirable to let the users rest the other fingers back 
onto the surface after typematic has initiated 804 and while 
typematic continues, but the user must do so without tap- 
ping. Decision diamond 805 causes typematic to be canceled 
and the typematic element deleted 778 if the user asynchro- 
nously taps another finger on the surface as if trying to hit 
another key. If this does not occur, decision diamond 782 
will eventually cause deletion of the typematic element 
when its finger lifts off. 

The typing recognition process described above thus 
allows the multi-touch surface to ergonomically emulate 
both the typing and hand resting capabilities of a standard 
mechanical keyboard. Crisp taps or impulsive presses on the 
surface generate key symbols as soon as the finger is 
released or decision diamond 792 verifies the impulse has 
peaked, ensuring prompt feedback to the user. Fingers 
intended to rest on the surface generate no keys as long as 
they are members of a synchronized finger press or release ' 
subset or are placed on the surface gently and remain there 
along with other fingers for a second or two. Once resting, 
fingers can be lifted and tapped or impulsively pressed. on 
the surface to generate key symbols without having to lift 
other resting fingers. Typematic is initiated ether by impul- 
sively pressing and maintaining distinguishable force on a 
key, or by holding a finger on a key while other fingers on 
the hand are lifted. Glancing motions of single fingers as 
they tap key regions are easily tolerated since most cursor 
manipulation must be initiated by synchronized slides of two 
or more fingers. 

Other embodiments of the invention will be apparent to 
those skilled in the art from consideration of the specifica- 
tion and practice of the invention disclosed herein. It is 
intended that the specification and examples be considered 
as exemplary only, with a true scope and spirit of the 
invention being indicated by the following claims. 

What is claimed is: 

1. A sensing device that is sensitive to changes in self- 
capacitance brought about by changes in proximity of a 
touch device to the sensing device, the sensing device 
comprising: 

two electrical switching means connected together in 
series having a common node, an input node, and an 
output node; 

a dielectric-covered sensing electrode connected to the 
common node between the two switching means; 

a power supply providing an approximately constant 
voltage connected to the input node of the series- 
connected switching means; 
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an integrating capacitor to accumulate charge transferred 
during multiple consecutive switching of the series 
connected switching means; 

another switching means connected in parallel across the 
integrating capacitor to deplete its residual charge; and 5 

a voltage-to-voltage translation device connected to the 
output node of the series-connected switching means 
which produces a voltage representative of the prox- 
imity of the touch device to the sensing device. 

2. The sensing device of claim 1, wherein the electrical 10 
switching means comprise semiconductor transistors that 
are switched on and off by dedicated control circuitry. 

3. By The sensing device of claim 1, wherein the electrical 
switching means comprise polymer transistors that are 
switched on and off by dedicated control circuitry. 15 

4. The sensing device of claim 1, wherein the electrical 
switching means comprise thin film transistors that are 
switched on and off by dedicated control circuitry. 

5. A sensing device that is sensitive to changes in self- 
capacitance brought about by changes in proximity of a 20 
touch device to the sensing device, the sensing device 
comprising: 

two electrical switching means connected together in 
series having a common node, an input node, and an 
output node; 25 

a dielectric-covered sensing electrode connected to the 
common node between the two switching means; 

a power supply providing an approximately constant 
voltage connected to the input node of the series- 30 
connected switching means; and 

an integrating current-to-voltage translation device con- 
nected to the output node of the series connected 
switching means, the current-to-voltage translation 
device producing a voltage representative of the prox- 35 
imity of the touch device to the sensing device. 

6. A sensing device that is sensitive to changes in self- 
capacitance brought about by changes in proximity of a 
touch device to the sensing device, the sensing device 
comprising: - - - - - 40 

two electrical switching means connected together in 
series having a common node an input node, and an 
output node, 

a dielectric-covered sensing electrode connected to the 
common node between the two switching means; and 45 

a power supply providing an approximately constant 
voltage connected to the input node of the series- 
connected switching means. 

7. A multi-touch surface apparatus for detecting a spatial 
arrangement of multiple touch devices on or near the surface 50 
of the multi-touch apparatus comprising: 

one of a rigid or flexible surface; 

a two-dimensional array of the sensing devices of claim 1 
arranged on the surface with their output nodes con- 55 
nected together and sharing the same integrating 
capacitor, charge depletion switch, and voltage-to- 
voltage translation device; 

control circuitry for sequentially enabling each of the 
sensor devices; ^ 

voltage measurement circuitry to convert sensor data to a 
digital code; and 

circuitry for communicating the digital code to another 
electronic device. 

8. A plurality of the multi-touch surface apparatuses of 65 
claim 7 arranged in the shape of one of a cube, a sphere, or 
any other three dimensional shape. 



9. The multi-touch surface apparatus of claim 7, wherein 
the surface comprises a micro-dimensional surface. 

10. The multi-touch surface apparatus of claim 7 being 
one of fabricated on or integrated with a display device. 

11. The multi-touch surface apparatus of claim 10, 
wherein the display device comprises one of a liquid crystal 
display (LCD) or a light-emitting polymer display (LPD). 

12. A multi-layer cover apparatus for the multi-touch 
surface apparatus of claim 7, comprising: 

a compliant dielectric layer; 

a deformable conductive layer formed on the dielectric 
layer, the conductive layer being electrically coupled to 
the voltage or current measurement device; and 

a touch layer formed on the conductive layer. 

13. The multi-layer concr apparatus of claim 12, wherein 
the touch layer has a symbol set printed thereon that can be 
removed and replaced with an alternative symbol set 

14. The multi-touch surface apparatus of claim 7, wherein 
the apparatus is ergonomically arched. 

15. The multi-touch surface apparatus of claim 7, wherein 
the apparatus includes hand configuration visual indicators. 

16. The multi-touch surface apparatus of claim 7, wherein 
the apparatus includes hand configuration audio indicators. 

17. A multi-touch surface apparatus for detecting a spatial 
arrangement of multiple touch devices on or near the surface 
of the multi-touch apparatus comprising: 

one of a rigid or flexible surface; 

a two-dimensional array of the sensing devices of claim 2 
arranged on the surface with their output nodes con- 
nected together and sharing the same current-to-voltage 
translation device; 

control circuitry for sequentially enabling each of the 
sensor devices; 

voltage measurement circuitry to convert sensor data to a 
digital code; and 

circuitry for communicating the digital code to another 
electronic device. 

.18. A multi-touch surface apparatus for detecting a.spatial._ 
arrangement of multiple touch devices on or near the surface 
of the multi-touch apparatus comprising: 

one of a rigid or flexible surface; 

a plurality of two-dimensional arrays of the sensing 
devices of claim 1 arranged on the surface in groups 
wherein the sensing devices within one group have 
their output nodes connected to corresponding sensing 
devices within other groups and share the same inte- 
grating capacitor, charge depletion switch, and voltage- 
to-voltage translation circuitry; 

control circuitry for enabling a single sensor device from 
each two-dimensional array; 

means for selecting the sensor voltage data from each 
two-dimensional array; 

voltage measurement circuitry to convert sensor voltage 
data to a digital code; and 

circuitry for communicating the digital code to another 
electronic device. 

19. The multi-touch surface apparatus of claim 18, 
wherein the sensor voltage data selecting means comprises 
one of a multiplexing circuitry and a plurality of voltage 
measurement circuits. 

20. A multi-touch surface apparatus for detecting a spatial 
arrangement of multiple touch devices on or near the surface 
of the multi-touch apparatus comprising: 

one of a rigid or flexible surface; 
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a plurality of the sensing devices of claim 1 arranged on 
the surface; 

control circuitry for sequentially enabling each of the 

sensor devices; 
voltage measurement circuitry to convert sensor data to a 

digital code; and 
circuitry for communicating the digital code to another 

electronic device. 

21. A multi-touch surface apparatus for sensing diverse 
configurations and activities of touch devices and generating 
integrated manual input to one of an electronic or electro- 
mechanical device, the apparatus comprising: 

an array of the proximity sensing devices of claim 1; 
a dielectric cover having symbols printed thereon that 

represent action -to-be-taken when engaged by the 

touch devices; 

scanning means for forming digital proximity images 
from the array of sensing devices; 

calibrating means for removing background offsets from 
the proximity images; 

recognition means for interpreting the configurations and 
activities of the touch devices that make up the prox- 
imity images; 

processing means for generating input signals in response 
to particular touch device configurations and motions; 
and 

communication means for sending the input signals to the 
electronic or electro-mechanical device. 

22. The multi-touch surface apparatus of claim 21, 
wherein the symbols printed on the dielectric cover can be 
removed and replaced with an alternative symbol set 

23. The multi -touch surface apparatus of claim 21, 
wherein the dielectric cover has conductive fibers therein, 
the conductive fibers being oriented normal to the array of 
sensing devices for conducting the capacirive ettect of the 
touch devices on the array of sensing devices. 

24. A multi-layer cover apparatus for the multi-touch 
surface apparatus of claim 21, the multi-layer cover appa- 
ratus comprising: 

a compliant dielectric layer; 

a deformable conductive layer formed on the dielectric 
layer, the conductive layer being electrically coupled to 
the voltage or current measurement device; and 

a touch layer formed on the conductive layer. 

25. The multi-touch surface apparatus of claim 21, 
wherein the apparatus is economically arched. 

26. The multi-touch surface apparatus of claim 21, 
wherein the apparatus includes hand configuration visual 
indicators. 

27. The multi-touch surface apparatus of claim 21, 
wherein the apparatus includes hand configuration audio 
indicators. 

28. The multi-touch surface apparatus of claim 21, being 
one of fabricated on or integrated with a display device. 

29. The multi -touch surface apparatus of claim 28, 
wherein the display device comprises one of a liquid crystal 
display (LCD) or a Light-emitting polymer display (LPD). 

30. A multi-touch surface apparatus for sensing diverse 
configurations and activities of fingers and palms of one or 
more hands near the surface and generating integrated 
manual input to one of an electronic or electro-mechanical 
device, the apparatus comprising: 

an array of proximity sensing means embedded in the 
surface; 

scanning means for forming digital proximity images 
from the proximities measured by the sensing means; 
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image segmentation means for collecting into groups 
those proximity image pixels intensified by contact of 
the same distinguishable part of a hand; 

contact tracking means for parameterizing hand contact 
features and trajectories as the contacts move across 
successive proximity images; 

contact identification means for determining which hand 
and which part of the hand is causing each surface 
contact; 

synchronization detection means for identifying subsets 
of identified contacts which touchdown or liftoff the 
surface at approximately the same time, and for gen- 
erating command signals in response to synchronous 
taps of multiple fingers on the surface; 

typing recognition means for generating intended key 
symbols from asynchronous finger taps; 

motion component extraction means for compressing 
multiple degrees of freedom of multiple fingers into 
degrees of freedom common in two and three dimen- 
sional graphical manipulation; 

chord motion recognition means for generating one of 
command and cursor manipulation signals in response 
to motion in one or more extracted degrees of freedom 
by a selected combination of fingers; 

pen grip detection means for recognizing contact arrange- 
ments which resemble the configuration of the hand 
when gripping a peo, generating inking signals from 
motions of the inner fingers, and generating cursor 
manipulation signals from motions of the palms while 
the inner fingers are lifted; and 

communication means for sending the sensed configura- 
tions and activities of finger and palms to one of the 
electronic or electro-mechanical device. 

31. A multi- touch surface apparatus for sensing diverse 
configurations and activities of fingers and palms of one or 
more hands near the surface and generating integrated 
manual input to one of an electronic or electro-mechanical 
device, the apparatus comprising: 

an array of proximity sensing means embedded in the 
surface; 

scanning means for forming digital proximity images 

from proximities measured by the sensing means; 
contact segmentation means for collecting proximity 

image pixels caused by the same hand part into groups; 
contact tracking means for parameterizing hand contact 

features and trajectories as the contacts move across 

successive proximity images; 
contact identification means for determining which hand 

and which part of the hand is causing each surface 

contact; 

synchronization detection means for identifying subsets 
of identified contacts which touchdown or liftoff the 
surface at approximately the same time; 

typing recognition means for generating intended key 
symbols from asynchronous finger taps; 

motion component extraction means for compressing the 
dozens of degrees of freedom in motions of multiple 
fingers into the degrees of freedom common in two and 
three dimensional graphical manipulation; 

chord motion recognition means for generating, command 
or cursor manipulation signals in response to motion in 
one or more extracted degrees of freedom by a selected 
combination of fingers; and 

communication means for sending said generated input 
signals to the electronic or electro-mechanical device. 
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32. A multi-touch surface apparatus for sensing diverse 
configurations and activities of fingers and palms of one or 
more hands near the surface and generating integrated 
manual input to one of an electronic or electro-mechanical 
device, the apparatus comprising: 

an array of proximity sensing means embedded in the 
surface; 

scanning means for forming digital proximity images 

from proximities measured by the sensing means; 
contact segmentation means for collecting proximity 

image pixels caused by the same hand part into groups; 
contact tracking means for parameterizing hand contact 

features and trajectories as the contacts move across 

successive proximity images; 
contact identification means for determining which hand 

and which part of the hand is causing each surface 

contact; 

pen grip detection means for recognizing contact arrange- 
ments which resemble the configuration of the hand 
when gripping a pen, generating inking signals from 
motions of the inner fingers, and generating cursor 
manipulation signals from motions of the palms while 
the inner fingers are lifted; and 

communication means for sending said generated input 
signals to the electronic or electro-mechanical device. 

33. A method for tracking and identifying hand contacts 
in a sequence of proximity images in order to support 
interpretation of hand configurations and activities related to 
typing, multiple degree-of-freedom manipulation via 
chords, and handwriting, the method comprising the steps 
of: 

segmenting each proximity image into groups of elec- 
trodes which indicate significant proximity, each group 
representing proximity of a distinguishable hand part or 
other touch device; 

extracting total proximity, position, shape, size, and ori- 
entation parameters from each group of electrodes; 

tracking, group paths through successive proximity 
images including detection of path endpoints at contact 
touchdown and liftoff; 

computing velocity and filtered position vectors along 
each path; 

assigning a hand and finger identity to each contact path 
by incorporating relative path positions and velocities, 
individual contact features, and previous estimates of 
hand and finger positions; and 

maintaining estimates of hand and finger positions from 
trajectories of paths currently assigned to the fingers, 
wherein the estimates provide high level feedback to 
bias segmentations and identifications in future images. 

34. A method for filtering and segmenting hand contacts 
in a sequence of proximity images in order to support 
interpretation of various contact sizes, shapes, orientations, 
and spacings, the method comprising the steps of: 

creating a smoothed copy of the most recent proximity 
image; 

searching for pixels with locally maximum proximity in 

the smoothed proximity image; 
searching outward from each local maximum pixel for 

contact boundary pixels using boundary tests of pixel 

and neighboring pixel proximities which depend on 

properties of hand contacts expected in a segmentation 

region of the pixel; 
forming groups from those pixels surrounding each local 

maximum pixel up to and including the boundary 

pixels; 
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combining groups of pixels which partially overlap; 
extracting group positions and features by fitting an 

ellipse to each group of pixels; and 
updating positions of the segmentation regions of the 
5 pixels in response to further analysis of the position and 

features extracted from each group of pixels. 

35. The method of claim 34, wherein sloppy segmentation 
regions of the pixels include rectangular areas under 
expected palm locations where only proximity valleys 

10 between the palm heels or pixels near a background signal 
level act as boundaries, and wherein a remaining image 
portion is a strict segmentation region that establishes group 
boundaries at a directional proximity minima encountered in 
a direction of search outward from a given local maximum 

is pixel. 

36. The method of claim 34, wherein segmentation region 
rules of a hand touching the surface override segmentation 
region rules of a hand not contacting the surface. 

37. The method of claim 34, wherein verified properties 
20 of each group of pixels in previous images are fed back as 

estimated hand offsets to adjust alignment of the segmen- 
tation regions for the current image. 

38. A method for associating into paths those surface 
contacts from successive proximity images caused by the 

25 same hand part and detecting liftoff from and touchdown 
onto the surface by each hand part, the method comprising 
the steps of: 

predicting the current positions of hand parts from their 
velocity along existing paths; 
30 finding for each of a group of pixels in current proximity 
image the existing path with a closest predicted path 
position; 

finding for each existing path the pixel group whose 
centroid is closest to the predicted path position and 
35 whose centroid is within a path-dependent tracking 
radius; 

pairing each pixel group with its closest path if the pixel 

group is also the closest pixel group to the path; 
starting new paths for remaining unpaired pixel groups; 
deactivating paths which have no pairable pixel groups 

within the path-dependent tracking radius; and 
updating path parameters from the measured parameters 
of the pixel group paired with each path. 
4S 39. A method of computing hand and finger position 
offsets from the measured positions of individual hand 
contacts on a multi-touch surface for the purpose of biasing 
future hand contact identifications or morphing the key 
layout in an integrated manual input device, the method 
50 comprising the steps of: 

establishing fingertip, thumb, or palm identities for each 
contact; 

establishing an offset weighting for each contact; 
computing a hand position offset, wherein the offset is a 
55 weighted average of the difference between a measured 
position of each contact and a predetermined default 
position of the hand part which corresponds to an 
established identity of the contact; and 
computing a finger position offset by subtracting a pre- 
60 determined default position of an associated hand part 
of the contact and the hand position offset from a 
measured position of the contact. 
40. The method of claim 39, wherein conservative esti- 
mates of hand and finger position offsets are maintained 
65 across a succession of proximity images even when hand 
contact identifications become unreliable, the method com- 
prising the steps of: 
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computing the confidence in current contact identifica- 
tions from the amount of information available for the 
identifications; 

computing a weighted average of the individual hand 
contact velocities; 

predicting a current hand and finger onset from the 
previous offset estimates and the weighted average 
velocity; 

computing current hand and finger offsets from current 
contact identities and measured contact locations; 

setting the currently measured hand and finger offsets to 
zero if the hand has no contacts in the current image; 
and 

updating the hand and finger offset estimates to the 
weighted average of the predicted offsets and currently 
measured offsets, wherein the relative weighting given 
to the measured offsets increases in proportion to the 
confidence level in the current identifications. 

41. The method of claim 40, wherein a hand sliding to the 
opposite side of the surface eliminates the estimated position 
of a lifted hand by imposing a minimum separation between 
the estimated hand positions, and permitting the hand with 
a highest identification confidence override the estimated 
position of the other hand. 

42. A method for establishing identities of hand contacts 
on a multi-touch surface using relative contact positions and 
features, the method comprising the steps of: 

defining a template of hand part attractor points on the 
surface, the attractor points for each hand roughly 
forming a ring; 

computing a matrix of distances from each surface contact 
to each attractor point; 

weighting the distances between each surface contact and 
each attractor point according to how closely measured 
contact features such as proximity to a surface, shape, 
size, eccentricity, orientation, distance to nearest neigh- 
bor contact, and velocity match features typical of the 
hand part the attractor point represents; 

finding a one-to-one mapping of the surface contacts to 
the attractor points that minimizes a sum of distances 
between each surface contact and its corresponding 
attractor point; and 

recognizing particular hand configurations from the num- 
ber and features of surface contacts assigned to par- 
ticular subsets of the attractor points. 

43. The method of claim 42, wherein an attractor point 
which is un assigned to a real surface contact because there 
are less surface contacts than attractor points contributes 
nothing to the sum of distances. 

44. The method of claim 42, wherein the distance metric 
used for computing the distance from a surface contact to an 
attractor point is the squared Euclidean distance. 

45. The method of claim 42, wherein the attractor point 
positions correspond to the positions of hand part surface 
contacts measured when each hand is in a neutral posture. 

46. The method of claim 42, wherein forepalm contacts of 
the hands are recognized by adding attractor points near the 
center of the attractor point ring of each hand, and weighting 
the distances to the forepalm attractor points so that contacts 
are assigned to the forepalm attractor points only if the hand 
produces enough contacts to nearly fill the attractor ring. 

47. The method of claim 42, wherein the portion of an 
attractor template corresponding to parts of a particular hand 
is translated to remain centered on the last estimated position 
of that hand. 
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48. The method of claim 42, wherein a portion of the 
attractor points template corresponding to parts of a particu- 
lar hand is rotated and/or scaled to match a previous esti- 
mated orientation and size of that particular hand. 
5 49. The method of claim 42, wherein all of the surface 
contacts to be assigned are assumed to have come from the 
same hand and all of the attractor points represent parts of 
one hand. 

50. The method of claim 42, wherein an additional hand 
identification means restricts assignment of the surface 
contacts to those attractor points which the hand identifica- 
tion means assigned to a particular hand. 

51. The method of claim 42, wherein the measured 
distance from a contact to a thumb attractor point on a given 
hand is weighted so as to encourage assignment to the thumb 

15 attractor when the total contact proximity is greater than that 
of a typical fingertip but less than that of a typical palm heel. 

52. The method of claim 42, wherein the measured 
distance from a contact to a thumb or inner palm heel 
attractor point on a given hand is weighted so as to encour- 

20 age assignment to that attractor when the contact orientation 
approaches the expected slant of a thumb or inner palm heel 
on the given hand. 

53. The method of claim 42, wherein the measured 
distance from a contact to a palm heel attractor point on a 

25 gi ven Dan ^ is weighted so as to encourage assignment to the 
palm heel attractor when the measured contact width or ratio 
of total proximity to eccentricity exceeds that of a typical 
finger. 

54. The method of claim 42, wherein additional contact 
3Q and inter-contact features are incorporated during a verifi- 
cation step, the verification step comprising shifting assign- 
ments found in the attractor points minimization step to 
make the assignments more consistent with additional fea- 
ture tests used in the verification step. 

55. The method of claim 54, wherein the verification step 
35 checks horizontal position coordinates of contacts assigned 

to attractor points corresponding to fingertips to ensure they 
are in increasing order for right hand fingertips and in 
decreasing order for left hand fingertips. 

56. The method of claim 54, wherein the verification step 
40 includes a thumb verification step comprising the following 

sub-steps: 

rinding an innermost finger contact by searching for a 
contact assigned to a filled attractor point correspond- 
ing to an innermost finger; 
45 computing a thumb factor as a function of the innermost 
finger contact relative to other finger contacts; 
shifting the innermost finger contact to a thumb attractor 
point if the innermost finger contact is not already 
assigned to the thumb attractor point and the thumb 
50 factor is above a predetermined thumb threshold; and 
shifting the innermost finger contact to a fingertip attrac- 
tor point if the innermost finger contact is currently 
assigned to the thumb attractor point and the thumb 
factor is below the prefeterrained thumb threshold. 
55 57. The method of claim 56, wherein the thumb actor is 
high if the orientation and size of the innermost finger 
contact are greater than those of other finger contacts. 

58. The method of claim 56, wherein the thumb factor is 
high if the separation, angle, and velocity of the innermost 

60 finger contact relative to other finger contacts are in ranges 
unique to opposable thumb presence or motion. 

59. The method of claim 56, wherein the verification step 
is performed when one of a fingertip or thumb attractor point 
is left unfilled by the attractor points minimization step. 

65 60. A method for ordering surface contacts and establish- 
ing finger, thumb, and palm identities, the method compris- 
ing the steps of: 
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finding a shortest path connecting all of the contacts 

assumed to be from a given hand; 
passing through each contact once to form an ordered 

loop; 

finding an innermost contact in the ordered loop; 

determining whether the innermost contact is a thumb, 
fingertip, or palm contact from contact and inter- 
contact features of the innermost contact; and 

assigning thumb, fingertip, or palm identities to non- 
innermost contacts based upon the features of the 
contacts, assignment of the innermost contacts, vertical 
position relative to assigned contacts, and the loop 
ordering. 

61. An apparatus for distinguishing palm heel contacts 
from other types of hand contacts in a system for recogniz- 
ing hand activity on a multi-touch surface and generating 
input signals to a competing device therefrom, the apparatus 
comprising: 

means for finding the nearest neighbor contact of a given 
contact in a plane of the surface; and 

means for suppressing identification of the given contact 
as a palm heel contact it a neighbor contact exists and 
is closer to the given contact than the anatomical 
separation between inner and outer portions of a palm 
heel 

62. The apparatus of claim 61, wherein the finding means 
ignores contacts identified as forepalm hand contacts. 

63. An apparatus for distinguishing palm heel contacts 
from other types of hand contacts in a system for recogniz- 
ing hand activity on a multi-touch surface and generating 
input signals to a competing device therefrom, the apparatus 
comprising: 

means for measuring the total proximity, orientation, and 

eccentricity of all contacts; 
means for encouraging identification of a given contact as 

a palm heel contact if its ratio of total proximity to 

eccentricity is larger than for a typical fingertip contact; 

and 

means for encouraging identification of a given contact as 
a palm heel contact as its orientation approaches the 
expected slant of a palm heel. 

64. An apparatus for distinguishing thumb contacts from 
other types of band contacts in a system for recognizing 
hand activity on a multi-touch surface and generating input 
signals to a competing device therefrom, the apparatus 
comprising: 

means for measuring the size and orientation of all 
contacts; 

means for encouraging identification of a given contact as 

a thumb contact if its size is larger than a typical 

fingertip contact; 
means for discouraging identification of a given contact as 

a thumb contact if its size is larger than a typical thumb 

contact; and 

means for encouraging identification of a given contact as 
a thumb contact as its orientation approaches the 
expected slant of the thumb. 

65. A method for determining which hand causes each 
surface contact detected on a multi-touch surface so that 
input signals generated by hand activity on the surface can 
depend on the identity of the hand performing the activity 
and so that multiple hands can perform independent activi- 
ties on the surface simultaneously, the method comprising 
the steps of: 

defining a template of hand part attractor points on the 
surface, the attractor points for each band approxi- 
mately forming a ring; 
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generating partitions which divide the set of all surface 
contacts into left hand clusters and right hand clusters; 

assigning finger and palm identities to the contacts within 
each cluster; 

5 computing for each partition an assignment fitness mea- 
sure which represents the biomechanical consistency of 
the fit of contact clusters to their assigned attractor 
rings; 

choosing the partition which has the best assignment 
10 fitness measure as the partition containing the true 
contact identities; and 
recognizing each hand's configuration from the combi- 
nation of and features of surface contacts assigned 
within each attractor ring of the best partition. 
is 66. The method of claim 65, wherein the hand assign- 
ments are re-computed only for proximity images in which 
new hand contacts are stabilizing. 

67. The method of claim 65, wherein a reactivated path 
for a temporarily-removed contact regains its previous iden- 

20 

68. The method of claim 65, wherein each attractor point 
is placed at an expected position of a corresponding hand 
part. 

69. The method of claim 65, wherein the partition gen- 
25 erating step comprises the following sub-steps: 

constructing approximately vertical contours between 
each horizontally adjacent contact; and 

constructing a partition from each contour by tentatively 
assigning contacts which are positioned to the left of a 
30 contour to the left hand cluster and contacts to the right 
of a contour to the right hand cluster. 

70. The method of claim 69, wherein the hand assign- 
ments of previously identified contacts can be locked so to 
not depend on which side of the dividing contour the 

35 contacts lie, while assignments of new contacts still depend 
on which side of the contour they lie. 

71. The method of claim 65, wherein each attractor ring 
is translated, scaled and/or rotated to match previous posi- 
tion estimates for the hand corresponding to the attractor 

40 ring. 

72. The method of claim 65, wherein the attractor points 
within each ring are individually offset by previously esti- 
mated finger offsets. 

73. The method of claim 65, wherein the assignment 
45 fitness measure is a total cost computed as a weighted sum 

of distances from each contact to its assigned attractor point 
in the attractor ring of its assigned hand cluster, and wherein 
the best partition is the one with the lowest total cost. 

74. The method of claim 73, wherein the distances 
50 between each surface contact and each attractor point are 

weighted according to how closely measured contact 
features, such as proximity to the surface, shape, size, 
eccentricity, orientation, distance to nearest neighbor 
contact, and velocity, match features typical of the hand part 
55 the attractor point represents. 

75. The method of claim 73, wherein a separation between 
the innermost finger and the next innermost finger is com- 
pared with a separation between the outermost finger and 
next outermost finger to obtain a handedness weighting 

60 which increases the total cost for a hand as the outermost 
separation becomes larger than is biomechanically consis- 
tent. 

76. The method of claim 73, wherein a hand portion of the 
total cost is decreased by a cluster velocity weighting 

65 function when the average velocity of the contact cluster of 
the hand indicates the hand is returning to its associated side 
of the surface. 
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77. The method of claim 73, wherein a hand portion of the 
total cost is increased by a palm cohesion weighting function 
when the contacts assigned to the palms of a hand are 
scattered over an area larger than the anatomical size of an 
outstretched palm. 5 

78. The method of claim 73, wherein the total cost of a 
partition is increased by a interhand separation weighting 
when the measured separation between contacts tentatively 
assigned to opposite hand clusters indicates that the hands 
may be overlapping or close to touching. 

79. A method for integrally extracting multiple degrees of 
freedom of hand motion from sliding motions of two or 
more fingers of a hand across a multi-touch surface, one of 
the fingers preferably being the opposable thumb, the 
method comprising the steps of: 

tracking, across successive scans of the proximity sensor 15 
array the trajectories of individual hand parts on the 
surface; 

finding an innermost and an outermost finger contact from 
contacts identified as fingers on the given hand; 

computing a scaling velocity component from a change in 20 
a distance between the innermost and outermost finger 
contacts; 

computing a rotational velocity component from a change 
in a vector angle between the innermost and outermost 
finger contacts; 25 

computing a translation weighting for each contacting 
finger; 

computing translational velocity components in two 
dimensions from a translation weighted average of the 3(J 
finger velocities tangential to surface; 

suppressively filtering components whose speeds are con- 
sistently lower than the fastest components; 

transmitting the filtered velocity components as control 
signals to an electronic or electro-mechanical device. 35 

80. The method of claim 79, wherein the scaling velocity 
computed from a change in distance between the innermost 
and outermost finger contacts is supplemented with a mea- 
sure of scaling velocity selective for symmetric scaling 
about a fixed point between the thumb and other fingers. . 40 

81. The method of claim 79, wherein the rotational 
velocity computed from a change in vector angle between 
the innermost and outermost finger contacts is supplemented 
with a measure of rotational velocity selective for symmetric 
rotational about a fixed point between the thumb and other 45 
fingers. 

82. The method of claim 79, wherein the translation 
weightings of the innermost and outermost fingers are 
constant but the translation weightings of central fingers are 
inversely related to polar component speeds so as to prevent 50 
vertical translation bias while performing hand scaling and 
rotation but otherwise include all available fingers in the 
translation average. 

83. The method of claim 79, wherein the translational 
weightings are related to the ratio of each finger's speed to 55 
the speed of the fastest finger so that if the user chooses to 
move fewer fingers than are on the surface the gain between 
individual finger motion and cursor motion does not 
decrease. 

84. The method of claim 79, wherein the suppressive ^ 
filtering step comprises the following two sub steps: 

downscaling each velocity component in proportion to a 
function of its average speed compared to the other 
average component speeds; 

dead-zone filtering each downscaled velocity component 65 
wherein the width of the dead -zone depends on the 
distribution of the current component speeds. 



85. The method of claim 79, wherein the orientation of an 
ellipse fitted to the thumb contact after each successive 
sensor array scan is transmitted as an additional degree of 
freedom control signal. 

86. A method for integrally extracting roll and tilt degrees 
of freedom of hand motion from pressure changes of three 
or more non-collinear hand contacts comprising any of 
thumbs, fingertips or palms, the method comprising the steps 
of: 

tracking across successive proximity images the trajecto- 
ries of individual hand parts on the surface; 

measuring proximities from each hand contact in a cali- 
bration proximity image once all available hand con- 
tacts have been stabilized; 

computing an average hand contact position from a post- 
calibration proximity image, wherein all hand contacts 
are weighted equally; 

computing a weighted average hand contact position from 
a post-calibration proximity image, wherein each hand 
contact is weighted according to the ratio of its current 
proximity to its calibrated proximity; 

computing for each post-calibration proximity image the 
difference vector between the weighted average hand 
contact position and the average hand contact position; 

dead-zone filtering the difference vector to remove varia- 
tions in proximity due to unintentional posture shifts; 
and 

transmitting the filtered difference vector from each post- 
calibration proximity image as roll and tilt control 
signals to an electronic or electro-mechanical device. 

87. The method of claim 86, wherein reference proximi- 
ties slowly adapt to decreases in individual contact proxim- 
ity. 

88. The method of claim 86, wherein the ratio of current 
hand contact proximity to its calibrated hand contact prox- 
imity is clipped to be greater than or equal to one. 

89. The method of claim 86, wherein an additional total 
hand proximity component is computed from the average of 
all current hand contact proximity to calibrated contact 
proximity ratios, and the total hand proximity component is 
transmitted to computing device. 

90. The method of claim 86, wherein the computation and 
transmission of hand roll and tilt rotational axes are initial- 
ized by resting all five fingers on the surface, tapping palms 
on the surface, and then resting palms on the surface. 

91. A manual input integration method for supporting 
diverse hand input activities such as resting the hands, 
typing, multiple degree-of- freedom manipulation, command 
gesturing and handwriting on a multi-touch surface, the 
method enabling users to instantaneously switch between 
the input activities by placing their hands in different con- 
figurations comprising distinguishable combinations of rela- 
tive hand contact timing, proximity, shape, size, position, 
motion and/or identity across a succession of surface prox- 
imity images, the method comprising the steps of: 

tracking each touching hand part across successive prox- 
imity images; 

measuring the times when each hand part touches down 
and lifts off the surface; 

detecting when hand parts touch down or lift off simul- 
taneously; 

producing discrete key symbols when the user asynchro- 
nously taps, holds, or slides a finger on key regions 
defined on the surface; 

producing discrete mouse button click commands, key 
commands, or no signals when the user synchronously 
taps two or more fingers from the same hand on the 
surface; 
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producing gesture commands or multiple degree-of- 
freedom manipulation signals when the user slides two 
or more fingers across the surface; and 

sending the produced symbols, commands and manipu- 
lation signals as input to an electronic or an electro- 5 
mechanical device. 

92. The method of claim 91, wherein production of 
discrete key symbols or mouse button click commands from 
single finger taps or finger chord taps is accompanied by 
transmission of activation signals to a light or sound feed- 
back generating device. 10 

93. The method of claim 91, wherein accidental synchro- 
nous finger taps during typing are prevented from disrupting 
the typing session by not producing input signals to a 
computing device when the accidental chord tap is the first 
detected chord tap since the last detected asynchronous key 15 
tap, the chord tap occurs within a typing timeout interval 
subsequent to the last detected asynchronous key tap, and no 
finger slides have been detected between the last detected 
asynchronous key tap and said accidental chord tap. 

94. The method of claim 91, wherein hand resting is 
tolerated by suppressing a generation of output commands 20 
when all or nearly all of the fingers simultaneously engage 
the multi-touch surface and remain substantially stationary. 

95. The method of claim 91, wherein user handwriting 
activity is distinguished from other input activities by its 
unique pen grip hand configuration with the following 25 
additional steps: 

establishing the finger or palm identity of each surface 
contact; 

measuring the relative positions and proximities of the 
identified contacts to determine whether the inner fin- 30 
gers are pinched while the outer fingers curl under the 
palm exposing their knuckles to rest on the surface; 

entering a handwriting mode for the hand if the above 
unique finger arrangement is detected; 

producing inking signals from the motions of the inner 35 
fingers on the surface while in handwriting mode; 

producing stylus lift signals each time the inner fingers lift 
off the surface while in handwriting mode; 

sending the inking signals to an electronic device for 
capture, display, or recognition; and 40 

leaving the handwriting mode after the hand has lifted and 
remained off the surface for a substantial time or if a 
non -pinched finger configuration is measured. 

96. The method of claim 95, wherein while in handwriting 
mode but the inner fingers are lifted, sliding and tapping 45 
motions of the palm heels produce cursor manipulation and 
clicking signals which are sent to the electronic device. 

97. The method of claim 95, wherein a stylus held 
between the pinched fingers touches the surface instead of 
the pinched fingers themselves to indicate pinch 50 
configuration, and wherein inking signals are measured from 
motion of the stylus. 

98. The method of claim 91, wherein the layout of key 
regions defined on the surface is morphed to fit the user's 
hand size and current position, the method comprising the 55 
following steps: 

defining a default key layout whose home row key regions 
lie roughly at predetermined default positions of the 
fingertips; 

identifying what hand part each surface contact comes 53 
from; 

detecting a layout homing gesture when all five fingers of 
a hand are placed on the surface in a partially closed 
posture; 

measuring during the layout homing gesture the position 65 
offsets of the homing hand and fingers with respect to 
the predetermined default finger positions; 
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translating the key regions normally typed by the hand by 
the measured hand and finger offsets such that each 
home row key region lies at approximately the mea- 
sured position of its corresponding finger; and 

updating the displayed positions of the key region sym- 
bols on a visual display embedded in the surface. 

99. The method of claim 91, wherein the layout of key 
regions defined on the surface is morphed to fit the user's 
hand size and current position, the method comprising the 
following steps: 

identifying what hand part each surface contact comes 
from; 

detecting a layout homing gesture when all five fingers of 
a hand are placed on the surface in a partially closed 
posture; 

measuring during the layout homing gesture the position 
of each finger on the surface; 

translating each home row key region and its neighboring 
keys by an amount such that the new position of the 
home row key region is approximately the same as the 
measured position of its corresponding finger; and 

updating the displayed positions of the key region sym- 
bols on a visual display embedded in the surface. 

100. The method of claim 91, wherein typing while the 
fingers mostly rest on the surface is made easier by not 
requiring finger liftoff quickly following each press of a key 
region, the method comprising the following steps: 

measuring the relative impulsiveness or forcefulness of 
finger touchdowns; 

producing key symbols from liftoff and impulsive or 
forceful touchdown of a finger while most fingers on 
the same hand are resting on the surface even if the 
finger continues to rest on the surface without quickly 
lifting back off the surface; and 

not producing key symbols when finger touchdowns are 
gentle or synchronous with other fingers. 

101. The method of claim 91, wherein typematic or 
automatic key repetition when a finger is held on a key is 
emulated "despite the fact that fingers which stay on the 
surface for extended periods are normally ignored to support 
hand resting, the method comprising the steps of: 

issuing a first keypress signal after a holding finger has 
touched down and remained on a desired key region for 
at least a hold setup time interval and all other fingers 
on same hand leave the surface within a release setup 
time after holding finger touched down; 

periodically issuing additional keypress signals every 
repeat time interval subsequent to the second keypress 
signal as long as the holding finger continues touching 
the desired key region; and 

ceasing repetitive issuance of the additional keypress 
signals when the holding finger lifts off the surface. 

102. The method of claim 101, wherein touchdown, 
resting or liftoff of hand contacts identified as palms on 
either hand does not affect the typematic state. 

103. The method of claim 101, wherein the cycle of 
keypress signal generation continues irrespective of whether 
other fingers touch down and rest on the surface subsequent 
to issuing the first keypress signal. 

104. The method of claim 101, wherein the repeat time 
interval is continuously adjusted to be inversely proportional 
to current measurements of holding finger proximity or 
pressure. 

105. A method for choosing what kinds of input signals 
will be generated and sent to an electronic or electro- 
mechanical device in response to tapping or sliding of 
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fingers on a multi-touch surface, the method comprising the 
following steps: 

identifying each contact on the surface as either a thumb, 
fingertip or palm; 

measuring the times when each hand part touches down 5 
and lifts off the surface; 

forming a set of those fingers which touch down from the 
all finger floating state before any one of the fingers lifts 
back off the surface; 

choosing the kinds of input signals to be generated by 
further distinctive motion of the fingers from the com- 
bination of finger identities in the set; 

generating input signals of this kind when further distinc- 
tive motions of the fingers occur; 15 

forming a subset any two or more fingers which touch 
down synchronously after at least one finger has lifted 
back off the surface; 

choosing a new kinds of input signals to be generated by 
further distinctive motion of the fingers from the com- 20 
bination of finger identities in the subset; 

generating input signals of this new land when further 
distinctive motions of the fingers occur; and 

continuing to form new subsets, choose and generate new 
kinds of input signals in response to liftoff and syn- 25 
chronous touchdowns until all fingers lift off the sur- 
face. 

106. The method of claim 105, wherein all sets or subsets 
which contain the same number of fingertips choose the 
same kinds of input signals, such that sets or subsets are 30 
uniquely distinguished by the number of fingertips they 
contain and whether they contain the thumb. 

107. The method of claim 105, wherein all sets or subsets 
which contain the same combination of thumb, index, 
middle, ring, and pinky fingers choose the same kinds of 35 
input signals. 

108. The method of claim 105, wherein the method is 
applied to contacts identified as left hand parts indepen- 
dently from contacts identified as right hand parts. 

109. The method of claim 105, wherein no input signals 
are generated after a set or subset is formed without further 40 
distinctive finger motions, to support resting of fingers on 
the surface. 

110. The method of claim 105, wherein one of the 
distinctive finger motions is synchronized liftoff of a ringer 
set or subset quickly following synchronized touchdown, 45 
and wherein each such motion generates a tap signal of the 
selected kind. 

111. The method of claim 105, wherein one of the 
distinctive finger motions is sliding of the finger set or subset 
across the surface, and wherein such motion continuously 50 
generates slide signals of the selected kind which include 
measurements of the sliding motion. 

112. The method of claim 105, wherein asynchronous 
touchdown quickly followed by liftoff of a finger forms a 
new subset of one finger and generates a tap event dependent 5S 
on the location on the surface of the touchdown. 

113. The method of claim 105, wherein generation of 
input signals is accompanied by generation of activation 
signals to a light or sound generating feedback device, and 
wherein the activation signals depend upon the kinds of 
input signals currently selected. 60 

114. The method of claim 105, wherein a new subset of 
fingers is formed upon simultaneous finger release from the 
all fingers resting state, wherein this new subset consists of 
those fingers which remain on the surface, and wherein this 
new subset chooses a new kinds of input signals which can 65 
be generated in response to further distinctive finger 
motions. 



115. A method for continuing generation of cursor move- 
ment or scrolling signals from a tangential motion of a touch 
device over a touch-sensitive input device surface after 
touch device liftoff from the surface if the touch device 
operator indicates that cursor movement continuation is 
desired by accelerating or failing to decelerate the tangential 
motion of the touch device before the touch device is lifted, 
the method comprising the following steps: 

measuring, storing and transmitting to a computing device 
two or more representative tangential velocities during 
touch device manipulation; 

computing and storing a liftoff velocity from touch device 
positions immediately prior to the touch device liftoff; 

comparing the liftoff velocity with the representative 
tangential velocities, and entering a mode for continu- 
ously moving the cursor if a tangential liftoff direction 
approximately equals the representative tangential 
directions and a tangential liftoff speed is greater than 
a predetermined fractional multiple of representative 
tangential speeds; 

continuously transmitting cursor movement signals after 
liftoff to a computing device such that the cursor 
movement velocity corresponds to one of the represen- 
tative tangential velocities; and 

ceasing transmission of the cursor movement signals 
when the touch device engages the surface again, if 
comparing means detects significant deceleration 
before liftoff, or if the computing device replies that the 
cursor can move no farther or a window can scroll no 
farther. 

116. The method of claim 115, wherein one of the 
representative tangential velocities is a weighted average of 
several instantaneous velocities. 

117. The method of claim 115, wherein the touch surface 
is a multi-touch surface, the touch devices are fingers, the 
cursor movement velocity is the hand translation, rotation, 
or scaling velocity extracted from the touching fingers, and 
the mode for continuously moving the cursor is entered 
when the velocity of the dominant hand motion component 
passes the deceleration test as the last fingers are lifted. 

118. A method for mapping gestures performed on a 
multi-touch surface by a right hand to simulate mouse 
manipulations, the method comprising the steps of: 

reserving asynchronous single-finger motions for typing 

and five-finger motions for hand resting; 
generating mouse pointer motion signals in response to 

translational slides of two fingertips; 
generating a single mouse click signal in response to a 

synchronized tap of two fingertips; 
generating mouse drag signals in response to translational 

slides of three fingertips; 
generating a double mouse click signal in response to a 

synchronized tap of three fingertips; 
generating window scrolling signals in response to trans- 
lational slides of four fingertips; 
generating a cut to clipboard signal in response to a 

pinching motion between the thumb and a fingertip; 
generating a copy to clipboard signal in response to a 

synchronized tap of the thumb and a fingertip; and 
generating a paste from clipboard signal in response to a 

movement of the thumb and a fingertip away from each 

other. 
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It is certified that error appears in the above-identified patent and that said Letters Patent is 
hereby corrected as shown below: 



Claim 1. 

Line 2, reads "during multiple consecutive switching of the series" and should read 

- during multiple consecutive switchings of the series — . 

Claim 3. 

Line 13, reads "By The sensing device of claim 1, wherein the electrical" and should 
read -- The sensing device of claim 1, wherein the electrical 

Claim 13. 

Line 15, reads "The multi-layer concr apparatus of claim 12, wherein" and should read 

- The multi-layer cover apparatus of claim 12, wherein 

Claim 3 L 

Line 60, reads "chord motion recognition means for generating, command" and should 
read chord motion recognition means for generating command --. 

Claim 56, 

Line 57, reads "The method of claim 56, wherein the thumb actor is" and should read 

- The method of claim 56, wherein the thumb factor is — . 

Claim 105. 

Line 18, reads "choosing a new kinds of input signals to be generated by" and should 
read -- choosing a new kind of input signals to be generated by 
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